Directly adding files to WebDav/FTP .dtCloud sync store

DEVONthink
1

To automate a bit more and rely less on cloud providers, I wanted to setup a WebDav sync store, then add files to it so that DEVONthink would pick them up on next sync

I’ve setup my inbox to sync to my new WebDav server (unencrypted), and it appears as xxx.dtCloud/inbox with master.plist and receipts/ content. However, sadly it doesn’t look like it’s actually possible to interact with this. Adding files to it doesn’t do anything,

So my question would be: Is it possible to add files directly to the DEVONthink sync store? If not, can we have something like this in the future? :slight_smile:

2

No it’s not possible, now or later.

As mentioned many, many times, a sync store contains raw, chunked, and optionally encrypted sync data about a database. This data is only useful to DEVONthink and DEVONthink To Go. It is not copying your files.
Also you should not be messing about in the internals of a sync store.

Add files to it by adding them to DEVONthink or DEVONthink To Go and syncing.

3 Likes
3

You could set up a folder on your NAS, mount it on your Mac and have DT move all files arriving there to their appropriate locations.

Tinkering with the sync folder is useless, pointless and might break the sync.

4

Thanks, that’s what I suspected. Too bad, because I feel there is a real usecase here. Right now DEVONthink can’t really integrate with most things (Zapier comes to mind on the server side), without using Siri Shortcuts or AppleScript somewhere in the middle (and Shortcuts doesn’t work so well either, because DTTGO File Provider isn’t pickable as a target, it’s greyed out)

For example, Prizmo or Genius Scan on iPhone are great PDF scanner apps, and both support automatic export to different locations like WebDav.
When on the same network, I can juse use a WebDav server on my Mac to drop the files into Inbox, but that of course doesn’t work when on the go.

Even with cloud solutions in the middle and exporting to Dropbox / iCloud Drive as interim, it still needs a Mac server somewhere that pulls those out and puts them into DEVOnthink

Just some stuff to think about. Some kind of automatic way to put files into DEVONthink would be nice to have

5

Yeah that’s exactly what I’m doing currently! Just would be nice to get rid of the server as central document-shoveler

6

I found a hacky solution that somewhat works on iOS

  1. Tell Prizmo (Genius Scan can’t do that yet) to auto-export to a local File folder (not iCloud or any other cloud provider - we want maximum privacy)
  2. Create a Siri Shortcut that iterates through all the files in that folder, adds them to DEVONthink and then deletes the file
  3. Glue the Siri Shortcut to an automation action, like “trigger whenever DEVONthink To Go is opened”

Shortcut: Shortcuts

For Shortcuts to be able to delete files automatically without confirmation, it needs to get enabled through Settings > Shortcuts > Advanced first. Only then it’s possible for individual shortcuts to enable ‘delete without asking’.

Not as nice as having a WebDAV/FTP server somewhere that just hooks into DEVONthink though

7

Why wouldn’t it? WebDAV protocol is not limited to local networks. You could either

  • forward the port to your server on your router and use that with a DynDNS setup to access your server from abroad (which is what I do for sync, btw), or
  • set up a VPN server on your WebDAV server, connect to it from abroad and thus effectively use the server as if it were on your local network

And, having APIs like AppleScript and JavaScript dictionaries is integration. In that respect, DT offers more than many Apple apps themselves.
But I suspect that you’re really addressing DTTG, i.e. the mobile version, here (though you posted in the DT subforum). On i*OS, everything is different, of course. And automation … well, you can always try your luck with Shortcuts. Or maybe Pythonista, or whatever. Everything is even messier there then on macOS.

8

In addition to @chrillek’s suggestion … as i have mentioned here more than once, while on the go away from Global HQ, I drop stuff into a Google drive in my phone. This is set to syncs with my Desktop. On Desktop I have a Hazel rule to watch this folder and on incoming move to the DEVONthink Global Inbox. From there DEVONthink rules and Auto classify take over.

Has worked for years and will continue to do so without trying to hack the sync location. KISS.

Or if iOS involved, rely on DEVONthink ToGo.

1 Like
9

Yeah this is just personal preference. If I’m putting so much focus on having encryption for all my sync stores, I want to avoid having unencrypted documents touch any cloud service altogether.
There’s data retention policies, Machine Learning model training, possible other scanning, etc etc

That’s why I’m jumping through the hoops to keep things local while I can :+1:

10

Again: Using your WebDAV server remotely through a VPN is as if your i*OS device were in your local network. And apart from setting it up once, it does not require jumping through any hoops: Just turn on VPN whenever you’re abroad. Which would be a save choice, anyway.

2 Likes
11

Surely I’m missing here something, because when in Mac, you can export/copy/move/drop your generated file into Inbox folder in Finder Side Bar, and in iOS/iPadOS you can a) Share and send into DTTG via Share option in original program or b) enable DTTG in Files App and write your file into whatever place inside DTTG folder structure in Files App.

The only warning here is open DTTG to allow sync once you’ve dropped the file via Share or via Files.

3 Likes
12

Yes I know I can do all of those things. The purpose of this thread was more to further see if automation can be improved. For example, if I could drop a file into my own WebDAV/FTP server (that’s running on a VPS somewhere), have that processed by a program and then directly inserted into the DEVONthink sync store would save me a bunch of share dialog actions into DTTGO. My files would just appear in my Inbox, the less manual actions, the better

Right now I am just letting it sync into a WebDAV server on my home network that’s hosted on my Mac, and those files are then getting moved into the DT Inbox folder so I’m more or less already doing what you suggest. It’s more: what to do from here, and what to do when I don’t have this home server running (I want to get rid of my mac mini eventually), etc.
I am relying on the mac server to act as a middle man with DEVONthink open 24/7, to pull files created with automations and move them into the inbox, and I’m not a fan of that, so I’m investigating how I can offload the tasks it is doing (not just dt) into a mini k8s cluster

1 Like
13

I interpret your “k8s” as “Kubernetes” (wow, the fear of typing has really come a long way since i18n). Which seems to be a lot of stuff to be throwing at … well, I must admit, I’m not quite sure what your problem is now.

Maybe you could take a step back and describe what it is that you’re actually trying to achieve, what your work situation is etc. It seems to me that we’re trying to solve the issue backwards here, since nobody but you can tell what you’re doing (or wanting to achieve).

1 Like
14

But you’re missing the point of the sync store. It’s for transferring sync data, not your files. It would require a fundamental modification to do what you’re suggesting with a sync store.

2 Likes
15

My usecase is a simple: Get files from stuff into DEVONthink secure, automatic and quick, without having a Mac running somewhere 24/7. That’s pretty much it

For example

  • Move files from document scanner apps and services - I know I can use the DTTGO share action, but when you scan a bunch of stuff daily, any extra step you can cut out is a win. Those apps can export into servers automatically
  • Connect services like Zapier - automatically generate PDFs of stuff, and get those into into the DT inbox. Not possible without a Mac downloading the files somewhere
  • Have digital office and postbox services (like https://www.dropscan.de/) that scan your letters move the files into DEVONthink. Those services support uploading into Dropbox/WebDAV/FTP, but how to get them out of Dropbox and into DT?

Again, the point is to not have a Mac somewhere doing the heavy lifting like pulling stuff out of Dropbox/WebDAV (that’s what I’m doing). I want to get rid of it eventually and move what it’s doing to a small VPS somewhere. :slight_smile:

So the tricky part is, how do you do that, when it’s not possible to add files to the sync storage directly? AppleScript is also only available on Mac so that shuts the automation options down to Siri Shortcuts.

For now I’m happy with Siri Shortcuts, the one I posted above works well enough, and makes sure the files aren’t uploaded to Cloud storage unencrypted. It already saved me a lot of time of manually moving documents on my iPhone around

16

See here

1 Like
17

Please forget about doing the impossible. The sync storage files are not your database and are not and never will be a target for what you are trying to do.

I now understand your “use case” but I don’t understand why it’s so essential to get stuff into DEVONthink running “somewhere else” on a computer which may or may not be powered-on. DEVONthink does not run anywhere where it is not installed. It is not running on the sync locations, for example. DEVONthink does not operate as a “client-server” model.

1 Like
18

Well for sure you cannot upload stuff to DT3 24/7 if you are not running DT3 24/7 - correct?

But you can do something almost as good pretty simply:

(1) Create a folder in Dropbox to be used for your uploads.

(2) Index your upload folder

(3) Create a smart rule which imports items from that folder into DT3

With this plan you need not run DT3 24/7.

3 Likes
19

I have pretty much the same in operation for a very long time, described above. Works. OP doesn’t like due to lack of encryption (see above). Oh well.

20

Now if we had something like this that also works on iOS :slight_smile: Sadly Siri Shortcuts can’t pull files out of Dropbox yet and I wasn’t able to find a cloud service that plays nicely with it either.
For now that means those documents are delayed importing until I turn on my Mac again, unless you know of something

Lack of encryption is easily solved by not using a cloud service and instead just a self-managed server as temporary step in the middle and just mounting that. Which again, is what I’m doing already anyway, but it still relies on my Mac for the heavy lifting, so I can’t use DTTGO by itself for quick data management.

Btw please don’t invalidate my usecases. There’s nothing wrong with me wanting e2e encryption or more automation, to solve a specific needs I have (that are different from your needs). A big reason why I use DEVONthink as such a central point of my setup is because it can sync with full encryption.

(My company also flat out forbids using unencrypted cloud storage for any kind of company documents)

anyway

warnings to not touch the sync store manually aside, I had some time at hand and started poking a new empty store with a single file synced out of curiosity to try and understand how DEVONthink sync actually works under the hood and it doesn’t seems so complex.

I’m sharing my notes in case anyone finds this useful in the future.

  • xxx.receipt is a folder representing a receipt
  • xxx.item within that receipt folder is a gzipped archive
  • within that gzipped archive is a binary list that represents the delta / file. To make it readable: plutil -convert xml1 xxxx

The delta for adding a new file looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>F162CA1C-611E-47ED-9702-ED79BD3E5E1A</key>
	<dict>
		<key>Added</key>
		<real>687340460.34072006</real>
		<key>Changed</key>
		<real>687340464.94158804</real>
		<key>CharCount</key>
		<integer>6</integer>
		<key>Created</key>
		<real>687340460.34075201</real>
		<key>FileSize</key>
		<integer>6</integer>
		<key>MEFiles</key>
		<data>
		MTIzYWJj
		</data>
		<key>Modified</key>
		<real>687340464.94096398</real>
		<key>Read</key>
		<real>687340464.93914902</real>
		<key>SyncAction</key>
		<integer>0</integer>
		<key>SyncParentIDs</key>
		<string>@ComeIn</string>
		<key>SyncSize</key>
		<integer>6</integer>
		<key>SyncXattr</key>
		<dict>
			<key>com.apple.TextEncoding</key>
			<data>
			dXRmLTg7MTM0MjE3OTg0
			</data>
		</dict>
		<key>Touched</key>
		<real>687340464.94096398</real>
		<key>UUID</key>
		<string>F162CA1C-611E-47ED-9702-ED79BD3E5E1A</string>
		<key>creator</key>
		<string>mac</string>
		<key>encoding</key>
		<integer>4</integer>
		<key>internalMetadata</key>
		<dict>
			<key>md_hash</key>
			<string>81335a7e63e4d2ed915c0ecaa9f6433a2ebe07f4</string>
			<key>md_language</key>
			<string>pl</string>
		</dict>
		<key>path</key>
		<string>~/Downloads/aaaa.dtBase2/Files.noindex/txt/1/xx.txt</string>
		<key>recordtype</key>
		<integer>1668577396</integer>
		<key>searchableText</key>
		<string>123abc</string>
		<key>selectionLength</key>
		<integer>0</integer>
		<key>selectionLocation</key>
		<integer>6</integer>
		<key>title</key>
		<string>xx</string>
		<key>type</key>
		<integer>3</integer>
		<key>version</key>
		<integer>30</integer>
		<key>visibleLength</key>
		<integer>6</integer>
		<key>visibleLocation</key>
		<integer>0</integer>
	</dict>
</dict>
</plist>

Document data is encoded directly into the binarylist under the data key

The things I wasn’t able to figure out were:

  • md_hash - it’s a sha1sum of something, no idea of what though. doesn’t seem to be a hash of the bytes or file content that got added
  • The filename (xxxx.item) is a sha256sum of something, not the binary list itself though
  • The name of the receipt folder is another sha256sum, but also not of the content of the folder

Then there is the xxx.manifest file, same thing: gzipped binarylist, contains an array with references to the individual receipts/items, like:

		<dict>
			<key>MFCreator</key>
			<string>mac</string>
			<key>MFDate</key>
			<real>687340473.68589997</real>
			<key>MFEntries</key>
			<array>
				<dict>
					<key>MEAction</key>
					<integer>2</integer>
					<key>MECRC</key>
					<integer>3560478890</integer>
					<key>MEDate</key>
					<real>687340533.47813594</real>
					<key>MENode</key>
					<true/>
					<key>MEPath</key>
					<string>/d96bb64ffbd49560144db3495eea6ab15a1766e47cc50b32c3fae524537805ce.receipt/4591b836b7ae9b8efd8c2eb8b12a21562717f972710cb352f16204cc5496d77c.item</string>
					<key>MESize</key>
					<integer>440</integer>
					<key>MEUUID</key>
					<string>1F84179F-8588-485E-8637-AA226FAD5FB3</string>
				</dict>
				<dict>
					<key>MECRC</key>
					<integer>2016335601</integer>
					<key>MEDate</key>
					<real>687340464.94096398</real>
					<key>MEPath</key>
					<string>/699e8447d69830fbbd4959b60159c945508fc6a157a9c19896c921b6b964c22f.receipt/2e04c1be352329781a02fff8c5a41bcaa3c5134b335819efecffb14d41ff4d14.item</string>
					<key>MESize</key>
					<integer>688</integer>
					<key>MEUUID</key>
					<string>F162CA1C-611E-47ED-9702-ED79BD3E5E1A</string>
				</dict>
			</array>
			<key>MFHost</key>
			<string>l60wyywrl1.local</string>
			<key>MFIndexedContents</key>
			<false/>
			<key>MFMaster</key>
			<string>765691B0E4A14B9BAA4190A6A5F159CE</string>
			<key>MFName</key>
			<string>687340473.686-8a195de683974c9784e0fa9ac5e2b4bd3250b25e-3795366903.manifest</string>
			<key>MFPrevious</key>
			<string>687339815.392-8a195de683974c9784e0fa9ac5e2b4bd3250b25e-2291163317.manifest</string>
			<key>MFReceiptUUID</key>
			<string>9D04B635-F4AA-472D-BAB1-9CCCA1E4569A</string>
		</dict>

Didn’t try to modify anything yet, but in theory, it should be possible to manually insert a receipt with the same format to tell DT that there is a new delta that hasn’t been applied to the database yet. Would be great to get some info to fill in the gaps about the format, but since this is not officially supported or advised, I doubt I’ll get anything :innocent:

I know, this is not recommended, but still interesting to try out if it’d actually work