To automate a bit more and rely less on cloud providers, I wanted to setup a WebDav sync store, then add files to it so that DEVONthink would pick them up on next sync
I’ve setup my inbox to sync to my new WebDav server (unencrypted), and it appears as xxx.dtCloud/inbox with master.plist and receipts/ content. However, sadly it doesn’t look like it’s actually possible to interact with this. Adding files to it doesn’t do anything,
So my question would be: Is it possible to add files directly to the DEVONthink sync store? If not, can we have something like this in the future?
As mentioned many, many times, a sync store contains raw, chunked, and optionally encrypted sync data about a database. This data is only useful to DEVONthink and DEVONthink To Go. It is not copying your files.
Also you should not be messing about in the internals of a sync store.
Add files to it by adding them to DEVONthink or DEVONthink To Go and syncing.
Thanks, that’s what I suspected. Too bad, because I feel there is a real usecase here. Right now DEVONthink can’t really integrate with most things (Zapier comes to mind on the server side), without using Siri Shortcuts or AppleScript somewhere in the middle (and Shortcuts doesn’t work so well either, because DTTGO File Provider isn’t pickable as a target, it’s greyed out)
For example, Prizmo or Genius Scan on iPhone are great PDF scanner apps, and both support automatic export to different locations like WebDav.
When on the same network, I can juse use a WebDav server on my Mac to drop the files into Inbox, but that of course doesn’t work when on the go.
Even with cloud solutions in the middle and exporting to Dropbox / iCloud Drive as interim, it still needs a Mac server somewhere that pulls those out and puts them into DEVOnthink
Just some stuff to think about. Some kind of automatic way to put files into DEVONthink would be nice to have
For Shortcuts to be able to delete files automatically without confirmation, it needs to get enabled through Settings > Shortcuts > Advanced first. Only then it’s possible for individual shortcuts to enable ‘delete without asking’.
Not as nice as having a WebDAV/FTP server somewhere that just hooks into DEVONthink though
Why wouldn’t it? WebDAV protocol is not limited to local networks. You could either
forward the port to your server on your router and use that with a DynDNS setup to access your server from abroad (which is what I do for sync, btw), or
set up a VPN server on your WebDAV server, connect to it from abroad and thus effectively use the server as if it were on your local network
But I suspect that you’re really addressing DTTG, i.e. the mobile version, here (though you posted in the DT subforum). On i*OS, everything is different, of course. And automation … well, you can always try your luck with Shortcuts. Or maybe Pythonista, or whatever. Everything is even messier there then on macOS.
In addition to @chrillek’s suggestion … as i have mentioned here more than once, while on the go away from Global HQ, I drop stuff into a Google drive in my phone. This is set to syncs with my Desktop. On Desktop I have a Hazel rule to watch this folder and on incoming move to the DEVONthink Global Inbox. From there DEVONthink rules and Auto classify take over.
Has worked for years and will continue to do so without trying to hack the sync location. KISS.
Yeah this is just personal preference. If I’m putting so much focus on having encryption for all my sync stores, I want to avoid having unencrypted documents touch any cloud service altogether.
There’s data retention policies, Machine Learning model training, possible other scanning, etc etc
That’s why I’m jumping through the hoops to keep things local while I can
Again: Using your WebDAV server remotely through a VPN is as if your i*OS device were in your local network. And apart from setting it up once, it does not require jumping through any hoops: Just turn on VPN whenever you’re abroad. Which would be a save choice, anyway.
Surely I’m missing here something, because when in Mac, you can export/copy/move/drop your generated file into Inbox folder in Finder Side Bar, and in iOS/iPadOS you can a) Share and send into DTTG via Share option in original program or b) enable DTTG in Files App and write your file into whatever place inside DTTG folder structure in Files App.
The only warning here is open DTTG to allow sync once you’ve dropped the file via Share or via Files.
Yes I know I can do all of those things. The purpose of this thread was more to further see if automation can be improved. For example, if I could drop a file into my own WebDAV/FTP server (that’s running on a VPS somewhere), have that processed by a program and then directly inserted into the DEVONthink sync store would save me a bunch of share dialog actions into DTTGO. My files would just appear in my Inbox, the less manual actions, the better
Right now I am just letting it sync into a WebDAV server on my home network that’s hosted on my Mac, and those files are then getting moved into the DT Inbox folder so I’m more or less already doing what you suggest. It’s more: what to do from here, and what to do when I don’t have this home server running (I want to get rid of my mac mini eventually), etc.
I am relying on the mac server to act as a middle man with DEVONthink open 24/7, to pull files created with automations and move them into the inbox, and I’m not a fan of that, so I’m investigating how I can offload the tasks it is doing (not just dt) into a mini k8s cluster
I interpret your “k8s” as “Kubernetes” (wow, the fear of typing has really come a long way since i18n). Which seems to be a lot of stuff to be throwing at … well, I must admit, I’m not quite sure what your problem is now.
Maybe you could take a step back and describe what it is that you’re actually trying to achieve, what your work situation is etc. It seems to me that we’re trying to solve the issue backwards here, since nobody but you can tell what you’re doing (or wanting to achieve).
My usecase is a simple: Get files from stuff into DEVONthink secure, automatic and quick, without having a Mac running somewhere 24/7. That’s pretty much it
Move files from document scanner apps and services - I know I can use the DTTGO share action, but when you scan a bunch of stuff daily, any extra step you can cut out is a win. Those apps can export into servers automatically
Connect services like Zapier - automatically generate PDFs of stuff, and get those into into the DT inbox. Not possible without a Mac downloading the files somewhere
Have digital office and postbox services (like https://www.dropscan.de/) that scan your letters move the files into DEVONthink. Those services support uploading into Dropbox/WebDAV/FTP, but how to get them out of Dropbox and into DT?
Again, the point is to not have a Mac somewhere doing the heavy lifting like pulling stuff out of Dropbox/WebDAV (that’s what I’m doing). I want to get rid of it eventually and move what it’s doing to a small VPS somewhere.
So the tricky part is, how do you do that, when it’s not possible to add files to the sync storage directly? AppleScript is also only available on Mac so that shuts the automation options down to Siri Shortcuts.
For now I’m happy with Siri Shortcuts, the one I posted above works well enough, and makes sure the files aren’t uploaded to Cloud storage unencrypted. It already saved me a lot of time of manually moving documents on my iPhone around
Please forget about doing the impossible. The sync storage files are not your database and are not and never will be a target for what you are trying to do.
I now understand your “use case” but I don’t understand why it’s so essential to get stuff into DEVONthink running “somewhere else” on a computer which may or may not be powered-on. DEVONthink does not run anywhere where it is not installed. It is not running on the sync locations, for example. DEVONthink does not operate as a “client-server” model.
Now if we had something like this that also works on iOS Sadly Siri Shortcuts can’t pull files out of Dropbox yet and I wasn’t able to find a cloud service that plays nicely with it either.
For now that means those documents are delayed importing until I turn on my Mac again, unless you know of something
Lack of encryption is easily solved by not using a cloud service and instead just a self-managed server as temporary step in the middle and just mounting that. Which again, is what I’m doing already anyway, but it still relies on my Mac for the heavy lifting, so I can’t use DTTGO by itself for quick data management.
Btw please don’t invalidate my usecases. There’s nothing wrong with me wanting e2e encryption or more automation, to solve a specific needs I have (that are different from your needs). A big reason why I use DEVONthink as such a central point of my setup is because it can sync with full encryption.
(My company also flat out forbids using unencrypted cloud storage for any kind of company documents)
warnings to not touch the sync store manually aside, I had some time at hand and started poking a new empty store with a single file synced out of curiosity to try and understand how DEVONthink sync actually works under the hood and it doesn’t seems so complex.
I’m sharing my notes in case anyone finds this useful in the future.
xxx.receipt is a folder representing a receipt
xxx.item within that receipt folder is a gzipped archive
within that gzipped archive is a binary list that represents the delta / file. To make it readable: plutil -convert xml1 xxxx
The delta for adding a new file looks something like this:
Didn’t try to modify anything yet, but in theory, it should be possible to manually insert a receipt with the same format to tell DT that there is a new delta that hasn’t been applied to the database yet. Would be great to get some info to fill in the gaps about the format, but since this is not officially supported or advised, I doubt I’ll get anything
I know, this is not recommended, but still interesting to try out if it’d actually work