Backup all open databases

PixelSnap 2022-06-09 at 22.37.53
Already I think I need to put this back at the top of the script.

set AppleScript's text item delimiters to od

Honestly don’t understand what it does, so I will look that up tomorrow or over the weekend to understand why both lines are there.

No real harm it seems in terms of file creation: Inbox 2022-06-09-07-33-46.dtBase2.zip
But it looks ugly in the Script Debugger log.

:+1:

One interesting question is: how often do you get the error. Only once – then it’s happening outside the repeat loop. Or once for every database – then it’s happening inside the repeat loop (unless you have only one database, that is). Than I’d have the script run in Script Debugger or Script Editor and see where exactly the error happens. I doubt that it’s the line with the do shell script. I tried that single line over here and it runs just fine. I guess that there’s a problem creating the file itself.

This is a good point.
I think new people coming to this product don’t have an easy way to get started with the scripting.
I mean, there’s not a nice control panel inside DEVONthink for scripting nor a little tutorial build into that scripting menu.

If I were to rework this for a beginner, I would

  1. Give the script menu a name (so that it can be automated with Keyboard Maestro)
  2. Create a getting started panel where you can go through and set up some basics that every new user will need. For example, they need to have some global variables defined such as selecting their backup drive. They need to see all the scripts and have a button to get help on each one, change the hotkey for each one, edit each one, and a button to create a new script.
  3. Create a forum post with a video if necessary for each script and link them in a way that allows a user to open the script menu and get help for all the common things.

Perhaps the scripts should be inside the DT3 system as a database and have a syntax highlighting system that allows the user to understand them, get help on them, organize them, duplicate them, and do so without ever modifying the original ones.

Seems nicer to do this than sending the user to finder to look at their scripts… especially since DEVONthink is already better than Finder, by far…

Now that we have this code outside of the repeat loop it is only once:

set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"

However, this error appears multiple times when I had the line inside the repeat loop.

That would break the de-facto standard on a Mac. I’m not even sure it’s possible given that the visibility of the script menu is controlled by script editor (which I find a horrible idea).

The rest of your points: scripting has nothing to do with DT in particular. It is available across macOS as a whole, in many places and apps. And three basics as well as the finer points are explained elsewhere (i posted a link to Apple’s documentation before). Repeating that in every scriptable app is pointless and tedious. And AppleScript is not the only scripting language (just the oldest one and the one with rhe worst string, date and array handing).

Personally, i do never use AppleScript if i can get the job done with JavaScript. Therefore, my suggestion about moving the set text delimiter stuff outside of the repeat loop might have been misleading.

1 Like

I was not at all sure I should muddy the waters even further but was sufficiently interested to have a working version of the script to suit my own purposes so spent some time on it.

Please note:

  1. This version suits my use. It may not suit the uses of others.
  2. I have removed all the credits at the start of the script—for which I apologise. I acknoweldge, however, all the help provided by others.
-- First set the path of the folder to which you wish to backup
property pbackup_path : "/Users/[your path]/Backups/"

tell application id "DNtp"
	
	set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"
	set all_databases to every database
	
	repeat with this_database in all_databases
		try
			set this_name to name of this_database
			
			-- We use the backup path property here
			set this_archive to pbackup_path & this_name & " " & this_date & ".dtBase2.zip"
			
			-- Set a general progress indicator
			show progress indicator "Weekly backup of all open databases…" steps count of all_databases
			
			with timeout of 3600 seconds
				
				-- Show which database we're processing
				step progress indicator this_name as string
				
				if (verify database this_database) is not 0 then error "Database --" & this_name & " is damaged."
				
				if not (optimize database this_database) then error "Optimisation --of database  " & this_name & "failed."
				
				if not (compress database this_database to this_archive) then --error "Backup  of database " & this_name & "failed."
				end if
			end timeout
			
		on error error_message number error_number
			hide progress indicator
			if the error_number is not -128 then log message "DEVONthink:" & error_message
		end try
	end repeat
	hide progress indicator
	
	-- Show an alert when it's all done
	display alert "Backups of all open databases complete."
end tell

Stephen (who now blocks his ears and closes his eyes for fear of retribution)

1 Like

Well, that caused me to scratch an itch. Here’s my take on it in JavaScript:

(() => {
  const app = Application("DEVONthink 3")
  app.includeStandardAdditions = true;
  const homeFolder = app.pathTo("home folder", {from: "user domain"}); 
  const pathToBackupFolder = `${homeFolder}/Backups/`
  const date = new Date();
  const dateString = date.toLocaleDateString("en-US", { year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute:'2-digit', second: '2-digit', hour12: false }).replaceAll(/[:\/ ]/g,"-").replace(",","");
  Progress.totalUnitCount = app.databases().length;
  Progress.title = 'Weekly backup of all open databases…';

  app.databases().forEach((db, index) => {
    const dbName = db.name();
    const archiveName = `${pathToBackupFolder}${dbName} ${dateString}.dtBase2.zip`;
    Progress.completedUnitCount = index+1;
    if (app.verify({database: db}) !== 0) {
      error(`Database '${dbName}' is damaged`);
    } else if (!app.optimize({database: db})) {
      error(`Optimisation of database '${dbName}' failed.`)
    } else if (!app.compress({database: db, to: archiveName})) {
      error(`Compression of database '${dbName}' failed.`)
    }
  })
  app.displayDialog("Database backup complete.")
})()

Contrary to my usual habit, this is actually tested (well, I let it run once).

It’s a tad more portable since it determines the user’s home folder dynamically (see the pathTo call at the top). The progress indicator is a very sorry excuse for one, Apple just didn’t care to provide anything useful for JXA. The date formatting is really weird, but I didn’t find a better way quickly to do that without recurring to the shell (which I usually do not want to do unless I have to). Edit See my post below for a better way.
Also, I decided to change the three if statements in the repeat loop to a single if … else if ... else if to make the structure clearer. The logic is the same, though. And I skipped the error handling, simply calling error and hoping that this will end up in DT’s log anyway.

And I didn’t add the timeout thingy. If that’s necessary, it should be something like
app.compress({database: db, to: archiveName}, { timeout: 3600 }). I.e. in JXA, the timeout is added to the method call as a final parameter object, not around a block. Why a scripting language would have a timeout parameter at all, is beyond me.

2 Likes

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

-- https://macscripter.net/viewtopic.php?id=24737
set this_time to replace_chars(time string of (current date), ":", "-")
tell (current date) to get "" & its year & "-" & (text -2 thru -1 of ("0" & (its month as integer) as text)) & "-" & its day
set this_date to result
set backup_date to this_date & "_" & this_time as string


-- https://macosxautomation.com/applescript/sbrt/sbrt-06.html
on replace_chars(this_text, search_string, replacement_string)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end replace_chars

This will remove the need to drop to a shell to get the date…
Perhaps the javascript idea is better…

I don’t like programming without an IDE that includes debugging and profiling tools.
What’s the best one for javascript?
No sense in buying Script Debugger which I still have on trial if AppleScript is so limiting and I can use something with JavaScript that does more.

Of course it is :crazy_face:
Instead of my original way to get a dateString

const dateString = date.toLocaleDateString("en-US", { year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute:'2-digit', second: '2-digit', hour12: false }).replaceAll(/[:\/ ]/g,"-").replace(",","");

I’ve now seen the light and suggest

const dateString = date.toISOString().replaceAll(/[:T]/g,"-").replace(/\.*$/,,"");```

date.toISOString returns something like 2011-10-05T14:48:00.000Z. The first replace call changes the colons and the T to dashes, the second one removes the point and everything after it. So it is a tiny bit more elegant than the AppleScript way, after all.

2 Likes

I’m sorely tempted to respond with an improved version of my AppleScript script (including the ability to cancel) but shall not because you will clearly win this little “contest”. :grin:

Stephen

1 Like

Elegant…? :thinking: :stuck_out_tongue:

1 Like
-- Daily backup archive - all open databases
-- Created by Christian Grunenberg on Mon Jun 22 2009.
-- Copyright (c) 2009-2022. All rights reserved.
-- addition for all open databases by coolgoose85
-- https://discourse.devontechnologies.com/t/backup-all-open-databases/50932

-- https://macscripter.net/viewtopic.php?id=24737
set this_time to replace_chars(time string of (current date), ":", "-")
tell (current date) to get "" & its year & "-" & (text -2 thru -1 of ("0" & (its month as integer) as text)) & "-" & its day
set this_date to result
set backup_date to this_date & " " & this_time as string

-- https://macosxautomation.com/applescript/sbrt/sbrt-06.html
on replace_chars(this_text, search_string, replacement_string)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end replace_chars

tell application id "DNtp"
	
	set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
	set all_databases to every database
	set AppleScript's text item delimiters to od
	
	repeat with this_database in all_databases
		try
			set this_path to path of this_database
			set this_name to name of this_database
			set this_archive to "/Volumes/YourVolumeName/Backup/" & this_name & " " & backup_date & ".dtBase2.zip" -- Change YourVolumeName to your external backup drive.
			
			show progress indicator "Daily Backup Archive of " & this_archive steps 3
			
			with timeout of 3600 seconds
				step progress indicator "Verifying database " & this_name & "..."
				if (verify database this_database) is not 0 then error "Database " & this_name & " is damaged."
				
				step progress indicator "Optimizing database " & this_name & "..."
				if not (optimize database this_database) then error "Optimization of database  " & this_name & "failed."
				
				step progress indicator "Zipping database " & this_name & "..."
				if not (compress database this_database to this_archive) then error "Backup of database " & this_name & "failed."
			end timeout
			
			hide progress indicator
		on error error_message number error_number
			hide progress indicator
			if the error_number is not -128 then log message "DEVONthink:" & error_message
		end try
	end repeat

Okay. I’ve run this a few times and it’s doing what I need it to do.
It no longer gives me permissions errors because it handles the dates with AppleScript instead of a shell.
Really helpful to simply open the databases I have changed and kick off the script after finishing work.

So thanks for all the tips and code ideas, guys. I appreciate it.

If you decide to use it, just be sure to edit the YourVolumeName in the script and set it to the name of your backup drive… I’ve been running the script from Script Debugger which gives me log I can check in the morning to see if any errors occurred.

I think that’s the only thing I might like to change is to change all the messaging commands to write to the DT3 log, but I haven’t looked at how to write to the DT3 log yet, so this is where I’m leaving it for now.

3 Likes

Noob question - isn’t Time Machine good enough?

Time Machine is better than no backups at all. It’s easy to set up and generally performs well enough. However, redundant backups of important data is always a good idea.

1 Like

imho Time Machine’s incremental backups are superior to full database copies
However, Time Machine data is stored locally; I also want to use cloud storage (offsite)
edit: for this backup, I use Arq Premium

In addition to Time Machine, I want a data backup disconnected from Devonthink.
i.e. ability to retrieve individual files without the complexity of retrieving the DT database
edit; for this backup, I use the export files&folder feature

Oh, I guess I have some homework to do. I didn’t realize you could back up the files within the DT databases as individual files without dealing with the DT database. I understand it’s easier with an index structure, but I need to explore how to do so when my data is stored within the DT database.

I’m not sure of @DTLow’s method here but we don’t suggest you mess about in the internals of the database package, in case you were headed that direction.

1 Like

I thought so but figured I was just behind the times with the latest intel as it relates to DT.

The script above doesn’t seem to work for me.

Does anyone have a script that will backup all open databases rather than doing it individually? I would like to set up a few databases as my primary grows, and would simplify the backup process.

The database archive function isn’t intended for incremental backups. They are best used as periodic full backups. Incrementals should be handled by your primary back strategy, e.g., Time Machine and external drives.