Script to export only new or modified files of all databases

[rfog]

What I want to accomplish is have all databases in an external disk, exported but accessible as if they were indexed. Why? Because in that situation I can access those files from Windows (for example). Indexing here is not an option.

I’m using the script described here as base:

Export entire database to filesystem for backup

Not necessary. Exactly. And again, no need to script if you have a macro or keyboard shortcut app such as KeyboardMaestro, Typinator, etc. Just turn the two DEVONthink shortcuts into one macro. Anyway, if you really want a script then use this. Use at your own risk. Tested to work here, but I don’t have your data and no support will be provided: use AppleScript version "2.4" -- Yosemite (10.10) or later use scripting additions tell application id "DNtp" set theDB to every record of …

From other messages, there is no option go across all open databases, then the script must be run manually for each database. Not a problem here. I could have an array of strings with the database names and loop over the arrays, or have one script for each database.

My main problem is how to check if destination record exists and is different, and do it only with files and not with tags or smart rules.

And, of course, being a seasoned C++ developer myself, AppleScript have completely alien concepts to me.

Any help is appreciated.

[chrillek]

• the databases have to be open before you can access them.
• looping over all of them should not be a problem.
• you can probably compare modification times of the record in the database and of the file on external disk.
• do you want to replicate database structure, too? Than you’d have to create folders on the disk for the database groups
• no need to use Applescript, JavaScript works with DT as well. See Introduction to JXA | JavaScript for Automation (JXA) for an introduction and some examples for DT. You’ll find others here in the forum.

Totally unrelated: is that really worth it? DT provides an export option already. That might be easier (and faster) than this scripting approach. Of course, if you’re working with 10s or 100s of gigabyte, always exporting all of them might be too slow.

[rfog]

@chrillek

• you can probably compare modification times of the record in the database and of the file on external disk.

One of my problems is I don’t know how to do it, not even what is “theResult” got from “to export…” in the script. Is it an object? A complete file? The file name only? How do I verify it is “AnyFile” and check if there is same file in “theDestination”. I’m completely null to AppleScript or even JS.

@chrillek

• do you want to replicate database structure, too? Than you’d have to create folders on the disk for the database groups

Yes, that is done automatically from the script. I’m assuming that “theResult” is the object or whatever it is with full path, as running the script as it is, it put the right file in the right folder in destination.

@chrillek

• no need to use Applescript, JavaScript works with DT as well. See Scripting with JXA | JavaScript for Automation (JXA) for an introduction and some examples for DT. You’ll find others here in the forum.

That is my big problem: all of that is completely alien to me.

@chrillek

Totally unrelated: is that really worth it? DT provides an export option already. That might be easier (and faster) than this scripting approach. Of course, of you’re working with 10s or 100s of gigabyte, always exporting all of them might be too slow.

I’m working with about 500 GB of files. I was using the normal menu Files and Folders export, manually from menu, and then export each database to the right base path (different from each database).

[chrillek]

I was hoping that I could avoid looking at that script… In order to make it easier to understand, I quote the relevant part here:

@korm

tell application id "DNtp"
	set theDB to every record of the current database
	set theDestination to "~/Downloads/DBE"
	repeat with thisItem in theDB
		set theResult to export record thisItem to theDestination
	end repeat
end tell

@rfog

One of my problems is I don’t know how to do it, not even what is “theResult” got from “to export…” in the script. Is it an object? A complete file?

Your first point of reference in this case should be the scripting dictionary for DT. Open the Script Editor, go to the “Data” menu and choose its fourth entry. Select DEVONThink 3 from the list of programs. Search for “export”… Well: the return value is of type text. No idea what it might contain (and why an export method even returns a value other than a boolean to indicate success or failure). Since the script quoted above ignores the return value (it just saves it in a variable that is never used again), why would you even care what it is?

Also, when you read the documentation on the export method in the function library, you’ll see that it unconditionally exports everything below its starting point. So when you call export on a database, there’s no chance to compare modification dates etc. before the fact. It’s an initial backup, not a delta one.

@rfog

That is my big problem: all of that is completely alien to me.

But as a “seasoned C++ programmer”, you should manage to read some basic text on another current programming language and get the gist of it, I guess. After all, JavaScript is a lot simpler than C++: No real inheritance, no templates, no operator overloading … And for JXA scripts, you do not even bother with object orientation at all: its all about arrays method calls. Why not give it a try?

@rfog

One of my problems is I don’t know how to do it,

One suggestion:

• First of all, start afresh. The script quoted before does not help you with what you want to do
• Second, loop over all databases
  • Inside the loop,
    • get the corresponding folder on your disk as a starting point (# 1)
    • loop over the content. That’ll iterate over all records in this database
    • now, for each record
      • if its type is group
        • check if a group with the name of this record exists in your current folder and create it if not
        • recurse to #1
      • Obviously, you have a normal record (i.e. no group). Get its name property
      • if a file with this name does not exist at the current location, create it and copy your current record to it (possibly using the record’s path property and the shell’s cp command)
      • else check if the record was modified more recently than the file (using the record’s modificationDate and the file’s last modify date, probably with a shell command)

As I said: Maybe it is easier to do a complete export of your database(s) in regular intervals

[DTLow]

@rfog

Only new or modified files …My main problem is how to check if destination record exists and is different, and do it only with files and not with tags or smart rules.

There’s also deleted files

It’s easy to check the record type and modification date; I use a smart rule

It’s simple code to check if a record/file exists (working with record name)
I wouldn’t even check for “different”, just replace the record/file

Gets complicated with groups (folders); new/moved/modified files

Also gets complicates if the record name changes; we’ve lost the link

what is “theResult” got from “to export…” in the script. Is it an object?

Just text; a simple file pathname

[BLUEFROG]

• Where is the destination in the Finder?

• I know you have massive databases. Do you have sufficient space available to support your info as exported data and also allow for future growth?

• Are you trying to maintain the group structures as well?

• Have you already exported your data to the external drive?
  If I’m understanding you correctly, I’d suggest using File > Export > files and Folders and do per-database exports to the destination first. This will remove an enormous burden off the script and allow it to handle the chore going forward.

[rfog]

Jim,

What I’m doing now is use the menu option to export each DB into a temporary folder, then cleanup the DEVONtech_storage files and then use FreeFileSync to sync with the final destination. No space problem as I have two 2 TB internal disks + one 2 TB USB3 attached in my 2017 iMac.

That answers most of your questions and I think that will be my way to do the things. I do it once a month and takes me about 1/2 hour to do all tasks.

Apart of that, I have two Smart Groups listing last month and week added and modified files.

Last one is go enjoy holidays and forget nerds like me for a time

[DTLow]

@rfog

I do it once a month and takes me about 1/2 hour to do all tasks.

Thanks for the hint about free file sync
I’ll add it to my toolbox

At ½ hour, I’d stick with this solution, running it more frequently,
even daily overnight

One point about folders vs tags; I prefer to use tags
I find exporting the tag folders gives a better structure export of my data

[rfog]

The script above exports tags, but empty inside (at least with my tag structure, that is completely un-hierarchical).

My idea was that: run it more frequently. What I do now, apart of the export, is TimeMachine in two of my three Mac, plus one specifically Databases folder with iDrive each night.

I really don’t want the export as a backup, but as a way to externally have access to those files. However, as BLUEFROG comments, the export command exports, does not handle the file for further examination. Solution could be export in a temporary folder and then script FreeFileSync to do the sync with the real destinations.

However, @DTLow be UTMOST CAREFUL with FreeFileSync (and other like-sync tools), as they copy iCloud Drive placeholders instead of real files when online-only files. They do well with OneDrive and Dropbox, but not with iCloud Drive, and developer says he cannot do any other thing because the API he is using.

[rfog]

@rfog

@DTLow be UTMOST CAREFUL with FreeFileSync (and other like-sync tools), as they copy iCloud Drive placeholders instead of real files when online-only files

I repeat it, to be completely clear against file disasters.