Backup all open databases

Hi,

I have modified existing script “Daily backup archive” which is located at ~/Library/Application Scripts/com.devon-technologies.think3/Menu/Export/ to go through all the open databases and create the backup.

It also create back with date and time so that it can be created multiple times in a day instead of just once in a day.

Feel free to review it and let me know if anyone find any issues.

– Daily backup archive.
– Created by Christian Grunenberg on Mon Jun 22 2009.
– Copyright (c) 2009-2019. All rights reserved.

tell application id “DNtp”
set allDatabases to every database
repeat with thisDatabase in allDatabases
try
if not (exists thisDatabase) then error “No database is open.”
set this_database to thisDatabase

        set this_date to do shell script "date +%Y-%m-%d-%H:%M:%S"
        
        set this_path to path of this_database
        set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
        set this_name to the last text item of this_path
        set AppleScript's text item delimiters to ""
        if this_name ends with ".dtBase2" then set this_name to (characters 1 thru -9 of this_name) as string
        set this_archive to "~/Backup/" & this_name & "_" & this_date & ".dtBase2.zip"
        set AppleScript's text item delimiters to od
        
        show progress indicator "Daily Backup Archive" steps 3
        
        with timeout of 1200 seconds
            step progress indicator "Verifying..."
            if (verify database this_database) is not 0 then error "Database is damaged."
            
            step progress indicator "Optimizing..."
            if not (optimize database this_database) then error "Optimization of database failed."
            
            step progress indicator "Zipping..."
            if not (compress database this_database to this_archive) then error "Backup failed."
        end timeout
        
        hide progress indicator
    on error error_message number error_number
        hide progress indicator
        if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
    end try
end repeat

end tell

It generates backup files as
Inbox_2019-10-11-09:35:16.dtBase2.zip
YetToBeOrganized_2019-10-11-09:36:26.dtBase2.zip

etc…

You can create a script file as
touch script_name.scpt
open script_name.scpt
paste above code and put it in
~/Library/Application Scripts/com.devon-technologies.think3/Menu/Export

DEVONthink will pick up the new script and will show in UI

2 Likes

Thanks for the script! Just a reminder that it’s not recommended to replace/modify default scripts, it’s recommended to modify copies to avoid conflicts after updates.

3 Likes

Very useful script - thanks

1 Like
-- Daily backup archive - all open databases
-- Created by Christian Grunenberg on Mon Jun 22 2009.
-- Copyright (c) 2009-2022. All rights reserved.
-- addition for all open databases by coolgoose85
-- https://discourse.devontechnologies.com/t/backup-all-open-databases/50932

tell application id "DNtp"
	set allDatabases to every database
	repeat with thisDatabase in allDatabases
		try
			if not (exists current database) then error "No database is open."
			set this_database to thisDatabase
			
			set this_date to do shell script "date +%Y-%m-%d-%H:%M:%S"
			
			set this_path to path of this_database
			set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
			set this_name to the last text item of this_path
			set AppleScript's text item delimiters to ""
			if this_name ends with ".dtBase2" then set this_name to (characters 1 thru -9 of this_name) as string
			set this_archive to "~/Backup/" & this_name & "_" & this_date & ".dtBase2.zip"
			set AppleScript's text item delimiters to od
			
			show progress indicator "Daily Backup Archive" steps 3
			
			with timeout of 1200 seconds
				step progress indicator "Verifying..."
				if (verify database this_database) is not 0 then error "Database is damaged."
				
				step progress indicator "Optimizing..."
				if not (optimize database this_database) then error "Optimization of database failed."
				
				step progress indicator "Zipping..."
				if not (compress database this_database to this_archive) then error "Backup failed."
			end timeout
			
			hide progress indicator
		on error error_message number error_number
			hide progress indicator
			if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
		end try
	end repeat
end tell

I’m only pasting again as code because I had to do some edits to fix the characters in the original post.
Gonna give it a try since it will let me run an overnight backup of everything I leave open, which sounds useful…

I have a question. What is the difference between running this archive system and simply shutting down DT3 and copying the databases folder?

Is there an advantage/disadvantage to doing this?
The script seems to freeze all action and it takes some time to complete.
So I would prefer a backup that doesn’t block continuing to work in the database or that can be done faster outside of DT3 with the database in a shutdown state.

1 Like

Because it uses the DT commands to “verify” and “optimize” the databases?..which said databases would have to be open to perform verification and optimization. These functions can’t be performed on shutdown/closed databases I believe.

Thanks for updated code…this looks really useful for archival backups.

1 Like

In my testing, it seems the “zipping” function is the one that “freezes” the action so maybe deleting that step, if the greater file sizes can be tolerated, might return control more quickly.

1 Like

Adding below info to make it complete for everyone.

You can create a script file as
touch script_name.scpt
open script_name.scpt
paste above code and put it in
~/Library/Application Scripts/com.devon-technologies.think3/Menu/Export

DEVONthink will pick up the new script and will show in UI

2 Likes

Why not simply open Applications/Utilities/Script Editor, paste and compile the code, then save it to the aforementioned folder?
:thinking: :slight_smile:

1 Like

Thanks for the responses. Quite helpful.

For me, this produced a file with a name of:

MySelectedDatabase_2022-06-09-00/15/18.dtBase2.zip

It placed it in ˜/Backup/

I’ll have to remove the / characters and figure out why it didn’t capture the other two databases that I had open.

			set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"

Interestingly, this line produces an error for me, and yet it works…

Error: a privilege violation occurred (errAEPrivilegeError:-10004)

I don’t know much about the Dictionary for DEVONthink, but I noticed that I can add it to the Script Editor Library

image

I ended up using Script Debugger instead of Script Editor to run the script. I added info to the status messages so that I could tell where I was at in the process.

			show progress indicator "Daily Backup Archive of " & this_archive steps 3
			
			with timeout of 3600 seconds
				step progress indicator "Verifying database " & this_name & "..."
				if (verify database this_database) is not 0 then error "Database " & this_name & " is damaged."
				
				step progress indicator "Optimizing database " & this_name & "..."
				if not (optimize database this_database) then error "Optimization of database  " & this_name & "failed."
				
				step progress indicator "Zipping database " & this_name & "..."
				if not (compress database this_database to this_archive) then error "Backup  of database " & this_name & "failed."
			end timeout

My second database took too long to backup.
1200 seconds isn’t enough. So I’ll triple this timeout parameter and try again tonight.


snip

Have you allowed all Automation requests for DEVONthink in System Preferences > Security & Privacy > Privacy?


And you’re getting the forward slashes because you’ve got colons in the time. Those are reserved characters and can’t be put in a filename.
Actually so are forward slashes, but macOS is allowing for it while caching the proper filename under-the-hood.
Take the colons out and strip out the time altogether.

1 Like

If they take out the time altogether, the colons should be gone in the process :wink:

@jsn Some remarks about what I’d do differently below. Just ignore them, if you wish. They’re mostly general remarks about coding style and habits, things I find helpful in obtaining clearer code. And clearer code is easier to understand as well as easier to debug and modify.

Move the set this date to do shell script "date +%Y-%m-%d" line before the repeat loop. Given that the date will not change while the backup is running (and even if it were, there’s no point in being overly accurate here) there’s no point in recalculating the same value over and over again. Generally, invariants should be moved out of loops to not obfuscate the logic.
Get rid of the set this_database to thisDatabase: What is the point of creating a variable with a slightly different name if you already have one (with a weird name, in my opinion, but that’s AppleScript lingo).
Move the line set {od, ... before the repeat loop. There’s no point of setting (and later resetting) the text delimiters if you only ever use the ‘/’ as a text delimiter inside the loop anyway. Same point as above with the do shell script line.
Analogously, move the line set AppleScript's text ... after the repeat loop.
And get rid of the set AppleScript's text item delimiter to "" completely - it makes no sense at all since you’re not splitting any strings after that point (and certainly not at an empty string).
In set this_name to (character 1 thru -9 of this_name) as string you could use text instead of character and thus get rid of the as string at the end. Again, that makes the code clearer (in my opinion), because taking a substring of a string should not require an explicit conversion to string. The concept is explained here Class Reference in the “Discussion” section for the Text class.

Then (you’ve found that out already the hard way): Using ~/Backup/ is not the best choice, since ~ is a shell shortcut for /Users/yourname/. Simply use /Users/yourname/Backup/ instead (replacing your name with your name, of course). There are other possibilities, too, but that one seems to be the simplest one in the given context.

I have doubts about the usefulness of display alert. If an error occurs, it will pop up the alert and then wait for user interaction. In the context of an automatically run script, this might not be ideal: I would not want my backup script to wait for user input in the middle of the night. You could alternatively use DT’s log message method that writes the messages to DTs log window. Also, error messages are a lot more useful if they do include the offending/broken object. Instead of "Database is damaged", you might want to use "Database " & this_database & " is damaged" and so on. You could pop up an alert at the end of the script informing the user that something went wrong and that they should inspect DT’s log.

2 Likes
-- Daily backup archive - all open databases
-- Created by Christian Grunenberg on Mon Jun 22 2009.
-- Copyright (c) 2009-2022. All rights reserved.
-- addition for all open databases by coolgoose85
-- https://discourse.devontechnologies.com/t/backup-all-open-databases/50932
-- Modifications courtesy of chrillek's advice: https://discourse.devontechnologies.com/t/backup-all-open-databases/50932/12

tell application id "DNtp"
	
	set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"
	set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
	set all_databases to every database
	
	repeat with this_database in all_databases
		try
			set this_path to path of this_database
			set this_name to the last text item of this_path
			if this_name ends with ".dtBase2" then set this_name to (characters 1 thru -9 of this_name) as string
			set this_archive to "/Volumes/YourBackupDriveName/Backup/" & this_name & " " & this_date & ".dtBase2.zip" -- was ˜Backup before, which created a folder under documents.
			
			show progress indicator "Daily Backup Archive of " & this_archive steps 3
			
			with timeout of 3600 seconds
				step progress indicator "Verifying database " & this_name & "..."
				if (verify database this_database) is not 0 then error "Database " & this_name & " is damaged."
				
				step progress indicator "Optimizing database " & this_name & "..."
				if not (optimize database this_database) then error "Optimization of database  " & this_name & "failed."
				
				step progress indicator "Zipping database " & this_name & "..."
				if not (compress database this_database to this_archive) then error "Backup  of database " & this_name & "failed."
			end timeout
			
			hide progress indicator
		on error error_message number error_number
			hide progress indicator
			if the error_number is not -128 then log message "DEVONthink:" & error_message
		end try
	end repeat
	
	set AppleScript's text item delimiters to od
end tell

Thanks for all of the tips.
I think I got everything but please if anybody sees a comment I missed or something I did wrong, correct it or let me know and I’ll correct it. I put the web links in the comments so that if anybody copies the script, they know where they got it and where they can go back to reference it.

I also changed the thisDatabase to this_database to stick with snake case convention.
And I changed the : to - in the time generation.

I did go through privacy parameters, but I think I’ve granted all the access that DEVONthink 3 requests in System Parameters, but I still have the error in my script. It’s an error that acts like a warning because the name is generated just fine. So perhaps I will see if there is a list of Privacy Parameters and make a visual guide to the System Parameters window. Wish all developers would make these parameter windows expandable and searchable with filters. Seems more logical to me if I could just type DEVONthink and then be shown all the settings related and check the boxes…

PixelSnap 2022-06-09 at 22.37.53
Already I think I need to put this back at the top of the script.

set AppleScript's text item delimiters to od

Honestly don’t understand what it does, so I will look that up tomorrow or over the weekend to understand why both lines are there.

No real harm it seems in terms of file creation: Inbox 2022-06-09-07-33-46.dtBase2.zip
But it looks ugly in the Script Debugger log.

:+1:

One interesting question is: how often do you get the error. Only once – then it’s happening outside the repeat loop. Or once for every database – then it’s happening inside the repeat loop (unless you have only one database, that is). Than I’d have the script run in Script Debugger or Script Editor and see where exactly the error happens. I doubt that it’s the line with the do shell script. I tried that single line over here and it runs just fine. I guess that there’s a problem creating the file itself.

This is a good point.
I think new people coming to this product don’t have an easy way to get started with the scripting.
I mean, there’s not a nice control panel inside DEVONthink for scripting nor a little tutorial build into that scripting menu.

If I were to rework this for a beginner, I would

  1. Give the script menu a name (so that it can be automated with Keyboard Maestro)
  2. Create a getting started panel where you can go through and set up some basics that every new user will need. For example, they need to have some global variables defined such as selecting their backup drive. They need to see all the scripts and have a button to get help on each one, change the hotkey for each one, edit each one, and a button to create a new script.
  3. Create a forum post with a video if necessary for each script and link them in a way that allows a user to open the script menu and get help for all the common things.

Perhaps the scripts should be inside the DT3 system as a database and have a syntax highlighting system that allows the user to understand them, get help on them, organize them, duplicate them, and do so without ever modifying the original ones.

Seems nicer to do this than sending the user to finder to look at their scripts… especially since DEVONthink is already better than Finder, by far…

Now that we have this code outside of the repeat loop it is only once:

set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"

However, this error appears multiple times when I had the line inside the repeat loop.

That would break the de-facto standard on a Mac. I’m not even sure it’s possible given that the visibility of the script menu is controlled by script editor (which I find a horrible idea).

The rest of your points: scripting has nothing to do with DT in particular. It is available across macOS as a whole, in many places and apps. And three basics as well as the finer points are explained elsewhere (i posted a link to Apple’s documentation before). Repeating that in every scriptable app is pointless and tedious. And AppleScript is not the only scripting language (just the oldest one and the one with rhe worst string, date and array handing).

Personally, i do never use AppleScript if i can get the job done with JavaScript. Therefore, my suggestion about moving the set text delimiter stuff outside of the repeat loop might have been misleading.

1 Like

I was not at all sure I should muddy the waters even further but was sufficiently interested to have a working version of the script to suit my own purposes so spent some time on it.

Please note:

  1. This version suits my use. It may not suit the uses of others.
  2. I have removed all the credits at the start of the script—for which I apologise. I acknoweldge, however, all the help provided by others.
-- First set the path of the folder to which you wish to backup
property pbackup_path : "/Users/[your path]/Backups/"

tell application id "DNtp"
	
	set this_date to do shell script "date +%Y-%m-%d-%H-%M-%S"
	set all_databases to every database
	
	repeat with this_database in all_databases
		try
			set this_name to name of this_database
			
			-- We use the backup path property here
			set this_archive to pbackup_path & this_name & " " & this_date & ".dtBase2.zip"
			
			-- Set a general progress indicator
			show progress indicator "Weekly backup of all open databases…" steps count of all_databases
			
			with timeout of 3600 seconds
				
				-- Show which database we're processing
				step progress indicator this_name as string
				
				if (verify database this_database) is not 0 then error "Database --" & this_name & " is damaged."
				
				if not (optimize database this_database) then error "Optimisation --of database  " & this_name & "failed."
				
				if not (compress database this_database to this_archive) then --error "Backup  of database " & this_name & "failed."
				end if
			end timeout
			
		on error error_message number error_number
			hide progress indicator
			if the error_number is not -128 then log message "DEVONthink:" & error_message
		end try
	end repeat
	hide progress indicator
	
	-- Show an alert when it's all done
	display alert "Backups of all open databases complete."
end tell

Stephen (who now blocks his ears and closes his eyes for fear of retribution)

1 Like

Well, that caused me to scratch an itch. Here’s my take on it in JavaScript:

(() => {
  const app = Application("DEVONthink 3")
  app.includeStandardAdditions = true;
  const homeFolder = app.pathTo("home folder", {from: "user domain"}); 
  const pathToBackupFolder = `${homeFolder}/Backups/`
  const date = new Date();
  const dateString = date.toLocaleDateString("en-US", { year: 'numeric', month: '2-digit', day: '2-digit', hour: '2-digit', minute:'2-digit', second: '2-digit', hour12: false }).replaceAll(/[:\/ ]/g,"-").replace(",","");
  Progress.totalUnitCount = app.databases().length;
  Progress.title = 'Weekly backup of all open databases…';

  app.databases().forEach((db, index) => {
    const dbName = db.name();
    const archiveName = `${pathToBackupFolder}${dbName} ${dateString}.dtBase2.zip`;
    Progress.completedUnitCount = index+1;
    if (app.verify({database: db}) !== 0) {
      error(`Database '${dbName}' is damaged`);
    } else if (!app.optimize({database: db})) {
      error(`Optimisation of database '${dbName}' failed.`)
    } else if (!app.compress({database: db, to: archiveName})) {
      error(`Compression of database '${dbName}' failed.`)
    }
  })
  app.displayDialog("Database backup complete.")
})()

Contrary to my usual habit, this is actually tested (well, I let it run once).

It’s a tad more portable since it determines the user’s home folder dynamically (see the pathTo call at the top). The progress indicator is a very sorry excuse for one, Apple just didn’t care to provide anything useful for JXA. The date formatting is really weird, but I didn’t find a better way quickly to do that without recurring to the shell (which I usually do not want to do unless I have to). Edit See my post below for a better way.
Also, I decided to change the three if statements in the repeat loop to a single if … else if ... else if to make the structure clearer. The logic is the same, though. And I skipped the error handling, simply calling error and hoping that this will end up in DT’s log anyway.

And I didn’t add the timeout thingy. If that’s necessary, it should be something like
app.compress({database: db, to: archiveName}, { timeout: 3600 }). I.e. in JXA, the timeout is added to the method call as a final parameter object, not around a block. Why a scripting language would have a timeout parameter at all, is beyond me.

2 Likes

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

-- https://macscripter.net/viewtopic.php?id=24737
set this_time to replace_chars(time string of (current date), ":", "-")
tell (current date) to get "" & its year & "-" & (text -2 thru -1 of ("0" & (its month as integer) as text)) & "-" & its day
set this_date to result
set backup_date to this_date & "_" & this_time as string


-- https://macosxautomation.com/applescript/sbrt/sbrt-06.html
on replace_chars(this_text, search_string, replacement_string)
	set AppleScript's text item delimiters to the search_string
	set the item_list to every text item of this_text
	set AppleScript's text item delimiters to the replacement_string
	set this_text to the item_list as string
	set AppleScript's text item delimiters to ""
	return this_text
end replace_chars

This will remove the need to drop to a shell to get the date…
Perhaps the javascript idea is better…

I don’t like programming without an IDE that includes debugging and profiling tools.
What’s the best one for javascript?
No sense in buying Script Debugger which I still have on trial if AppleScript is so limiting and I can use something with JavaScript that does more.