Backup a complete database in an DEVONthink agnostic way

I’m thinking of a yearly or backup strategy for my DEVONthink databases to a format which is agnostic of DEVONthink.

This means: I woud llike to export all documents of a database (or even of all databases in one go) to a plain filesystem folder structure with as much metadata as possible in the document files themselves.

This is only a last resort backup for when in 50 years DEVONthink became the new Google and “had to abandon” the old database format from the first half of the century and my son missed the migration deadline… :dizzy_face:

But seriously, it is good to have a way to backup DEVONthink databases into a form which does not rely on DEVONthink for later retrieval.

2 Likes

File > Export > Files & Folders… does exactly this. Just select all groups/documents of your database in the main view and then export them.

DEVONthink 3 is even able to convert databases of version 1.x :open_mouth:

3 Likes

As per @cgrunenberg, I use File>Export>Files&Folders as part of my weekly backups
This preserves folder, dates and tag metadata, and file comments

1 Like

Cool! Good tip to select multiple Groups in the database’s root view.

That’s cool too: DEVONthink is really looking after its users. :smile:

The ease with which I can get my data back out of DT is one of the reasons I chose DT in the first place. It’s ironic, of course, that I consider DT to be one of the best pieces of software I have ever come across and have not the slightest intention of separating it and my data :smiley:

4 Likes

Please bring this Feature to iOS/iPadOS! Currently it’s practically impossible to batch export data there.

This is already on our list for future improvements.

Perfect! Thank you! Really looking forward to the feature! :slight_smile:

When is it best to use Export>Files &Folders versus Export>Database Archive? Obviously F&F allows the selection of individual F&F’s but in the case of exporting a whole database, is there a reason to use one over the other?

Exporting a database archive creates an optimized ZIP file of the database as a secondary backup option. This can be decompressed directly to a database, unlike exporting files and folders.
However, it is not something meant to be used often, especially as databases grow. I’d suggest weekly or monthly at a maximum.

I use both tools in my weekly backups

As requested above, with Export>Files&Folders I have a backup that is “DT agnostic”
I can view speific files directly

Export>DatabaseArchive creates a backup to be used in restoring the database.
I would not access it to view specific files

Thanks to @BLUEFROG and @DTLow for clarifying.

You’re welcome.

How do you automate this process?

I have automated Export>Database archive on the DB level (i used a triggered KM cronjob).
But what about the above as it is not possible to select a DB directly?

I haven’t figured out automation; just a weekly manual operation

But what about the above as it is not possible to select a DB directly?

Database Archive is a good backup solution (not agnostic), but can’t be accessed directly to retrieve a specific file from the database
The backup has to unzipped and loaded into Devonthink

Here is how I do it … all in addition to routine backups with TimeMachine, rsync copies of this backups folder and others to an attached USB drive which itself is included in the BackBlaze backup.

I have this Apple script, based on (I think) from a Script provided by DEVONthink.

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

tell application "DEVONthink 3"
	set theDatabases to every database
	show progress indicator "Running Backup" steps (count of theDatabases)
	repeat with thisDatabase in theDatabases
		set this_database to thisDatabase
		step progress indicator (name of this_database as string)
		set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"
		set this_path to path of this_database
		set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
		set this_name to the last text item of this_path
		set AppleScript's text item delimiters to ""
		if this_name ends with ".dtBase2" then set this_name to (characters 1 thru -9 of this_name) as string
		set this_archive to "~/Backups/DEVONthink_Backups/" & this_name & " " & this_date & ".dtBase2.zip"
		set AppleScript's text item delimiters to od
		
		with timeout of 1200 seconds
			set thisMsg to "Verified: " & the name of thisDatabase & " with result "
			set thisResultV to verify database this_database
			set thisLog to log message (thisMsg & thisResultV & " errors and/or missing files")
			if thisResultV = 0 then
				set thisMsg to "Optimized: " & the name of thisDatabase & " with result "
				set thisResultO to optimize database this_database
				if not thisResultO then
					set thisLog to log message (thisMsg & " Optimization of database failed.")
				else
					set thisLog to log message (thisMsg & " Optimization of database success.")
				end if
				
				if thisResultO then
					set thisMsg to "Zipped: " & the name of thisDatabase & " with result "
					set thisResultC to (compress database this_database to this_archive)
					if not thisResultC then
						set thisLog to log message (thisMsg & " zipped of database failed.")
					else
						set thisLog to log message (thisMsg & " zipped of database success.")
					end if
				end if
			end if
		end timeout
	end repeat
	
	hide progress indicator
	
	display notification "DT backup finished."
	
end tell

This Apple script called by a bash command file i keep in ~\cmd folder. I have this command so that on a whim, I can just issue the command to take a backup then and there.

/usr/bin/osascript "/Users/rmschne/Library/Mobile Documents/com~apple~ScriptEditor2/Documents/DEVONthink Backup.scpt"

This command file is scheduled in crontab (standard and essential part of MacOS available to users also) to run three times a week at night when the machine is turned on and all my active databases are open in DEVONthink. I use “lingon x” for easy manipulation of cron’s crontab file. Or see crontab’s man file for details on how to schedule something.

The only manual intervention (I have a reminder in Things) to go clear out the excessive older backups from the files in ~\backups\devonthink_backups" folder. I haven’t yet figured out how to delete all but the last two files of a particular set. Will in due course.

An improvement might be to look for all DEVONthink databases, open, and then backup, but I figure if not open, then not important.

The benefit of these zip files in ~\backups on local disk is that they are backed up by the backup regime I mention at the top. Perhaps over the top, but there you go.

I’m sure the Apple script gurus here can improve this!

edit: Caveat emptor

and just thought of a way to schedule deletion of unwanted zip files. will do that tomorrow when fresh.

2 Likes

Thanks for sharing your approach.

So far I was doing the following:

  • local timemachine
  • daily / weekly Export>Database archive on a DB level (triggered by KM)
  • remote encrypted backup of that via arq into cloud storage (arq 5 > now arq 7, arq 6 for god’s sake I skipped as it was very buggy)

But I am not sure now what Export>Files&Folders would give me in addition of Export>Database archive? (I am fine that this one mega zip which i will need to unzip in case of disaster).

An example is the retrieval of a single file (record) or group that you accidentally corrupted

I see. Arq does that for me (it backups exact finder structure) so I guess I should be fine then with an overall zip archive backup strategy in addition.

I’m looking at Arq (not agnostic) for my offsite backup strategy
The “exact finder structure” can be a bit convoluted, but it provides a snapshot recovery for specific times