Backup: complete, and as automatic as possible

I’m still fairly new to DevonThink, and need some help on the perennial topic of backup.

I need to be able to back up all my DT databases (including the Global Inbox!) to an external drive, in as few steps as possible.

As I understand it now, the command: file–>export–>Database Archive only works on one database at a time. this is unwieldy.

the script “Export–>Daily Backup Archive” seems limited to putting the backup in a folder “backup” (where is it?) and does not allow one to change the backup location on the fly. This, too isn’t an optimal situation.

Is there another solution I have missed?

Can someone at least show me how to edit the script to insert a different backup location?

I note that I just suffered a hard drive crash. I had carefully backed-up my DT databases—but hadn’t known about the Global Inbox being in another location. The result was two weeks worth of clipping and storing gone. I need a comprehensive solution that won’t let me do something like this again. (And the disk failed just as I was doing a full disk-clone with Carbon Copy Cloner–I know about these types of solutions.)

thanks!

bb

I’d also like to know the suggestions of experienced DT users. But I personally have not used any of built-in DT backup functions, and instead I rely entirely on whole-disk backup solutions like TimeMachine and SuperDuper.

I also use a tool called Arq to backup directly to a bucket on S3. The advantages are that it is running continuously and backing up every hour, and the backups are on “the cloud”, which offer extra security in case of equipment theft. Oh, and I don’t have to remember to plug-in any external hard drive.

Of course it could get a little expensive if you decide to backup too much info to S3. But you could only backup the important and always changing stuff and rely on a backup to an external hard drive for the rest of your info.

To backup DT’s global inbox just make sure you are backing up the folder ~/Library/Application Support/DEVONthink Pro 2/Inbox.dtBase2

The ~ is your user’s home folder.

Over here, I use (1) Time Machine (as @jorgeg mentioned), (2) ChronoSync runs a daily job to clone my system disk (where most of my regularly-used databases reside in the Documents folder), (3) a second daily ChronoSync job to copy my secondary backup disk (remainder of my databases reside there) to a clone of that drive.

In other words, these automated, attendant-free, processes ensure that I have at least one mirror image of every data disk - two in the case of the system drive. Time Machine provides multiple generations, the other backups are just today-only. (I don’t use ChronoSync’s archiving feature, just because it takes up more space on a drive than it is worth to me.) I’m planning to add an additional, weekly, backup for rotating offsite storage (to my bank’s vault).

Finally, I regularly transfer copies of my professional databases to a laptop - not for backup purposes, but it does have that side-effect.

I don’t use the DEVONthink scripts you mention because my bots do all the work - more reliably than if I did it myself.

Congratulations for your concern about having good backups, and consolations for your loss of work from the hard drive failure that lost documents stored in the Global Inbox.

In the many years and on the many Macs I’ve worked with I’ve only experienced one hard drive crash. Because that drive had a very recent backup of my databases to an external drive, I didn’t lose any data.

But I’ve concluded that there is no such thing as a Really Comprehensive Backup Strategy that can also be Completely Automated. There are common situations such as the user’s decision to make major modifications to a database, where a manual backup procedure before making those modifications should be done.

Let’s start with the free automatic backup routine that’s included in OS X, Time Machine. I use it and recommend it. It can be scheduled to make hourly backups of incremental changes to the data on a computer. If you had been using Time Machine (or the closely related Time Capsule) backup system, it would have allowed recovery of your Global Inbox data after your hard drive crash.

There are two weaknesses in Time Machine backups of DEVONthink databases, however, and they should be taken into account in your backup strategy.

The most serious weakness of Time Machine is that it doesn’t check database integrity. If a database has become damaged, Time Machine’s backup is damaged.

That weakness can be mitigated by user intervention. If you get into the habit of manually invoking Tools > Verify & Repair periodically to check for errors, followed by Tools > Backup & Optimize if no errors are found, the trustworthiness of Time Machine backups is vastly greater.

A second weakness of Time Machine backups is that there’s a possibility that at the moment changes to the files in a database are copied to the backup, there might still be data in memory that hasn’t yet been saved to disk. Thus, there’s a possibility of missing data in the backup. This rarely turns out to be a serious problem, but it can happen. Again, user intervention by quitting the DEVONthink application will mitigate the potential problem. If you are working such that Time Machine is automatically backing up data every hour in tha background, I wouldn’t worry too much. Again, however, if one has run Verify & Repair once in a while during the day, followed by Backup & Optimize, the Tools > Restore Backup routine would correct errors in a database recovered from Time Machine that might result from this weakness of Time Machine, with little or no loss of data.

I do most of my database work on a laptop, so Time Machine isn’t doing hourly automatic backups. When I return my laptop to the desk and mount the external drive on which Time Machine backups are stored, I will first invoke Verify & Repair on each database that has been modified. My databases are very stable - it’s been years since an error was reported, but it’s prudent to do this. If no error was reported, than I invoke Backup & Optimize. Next, I quit the DEVONthink application and manually start a Time Machine backup. I’ve tested database backups created in this way, and they have all been good.

My backups of last resort, copies of which are stored offsite, are database archives produced by the File > Export > Database archive or by one of the Scripts > Export routines. They are important in my backup strategy, as they will be available if Bad Things such as theft of my computer equipment or a fire were to result in loss, including loss of the Time Machine backups. Database archives are the smallest possible, complete backups of databases. I copy them to a portable drive that’s stored in a safety deposit box at my bank. (I can’t consider cloud storage of my collection of database archives. It would take days to upload them, and days to retrieve them if needed, given my Internet access speeds.)

While it might be possible to automate production of database archives, it would be hard to avoid scripting timeout problems. The most time-consuming part of the routine is the compression phase - it may take quite a few minutes to zip a multi-gigabyte database. Another potential problem with automation would be that if an error is found in a database, the routine will stop with an error message report.

Every few weeks I update my database archives to a portable hard drive and swap it with the portable drive that’s currently at my bank. Usually, only a few of my collection of databases will require creation of a new archive during that time period - it is an easy task to compare modification dates of the existing archives to modification dates of the corresponding databases.

These are helpful replies, but I still didn’t see any answer to the OP’s question (and mine)

— how to change the default location of the Export > Daily Backup Archive?

I’d like this to go straight into my Dropbox folder rather than ~/Backup

Any tips?

I use Hazel to automate a copy from the Backup folder to another folder or disk.

I am still lost as there seems to be no way to change this location from within DT. Do I really need to buy more software to do something apparently so simple?

I’m using the script, “Daily backup archive” to create a zip-file archive of my whole database (I modded the script to put it in my Dropbox).

Does the archive script do the “Verify & Repair” and “Backup & Optimize” commands from the Tools menu before making the zip archive?

I’m not sure if I need to do those before I run the archive script.

The Archive routine checks for database errors and will stop and notify the user if an error is detected. That’s A Good Thing, as the user then can take steps to clean up the database, and a damaged backup wasn’t created.

I use a redundant backup strategy. I use Time Machine, and also create database archives that are stored offsite, in case of loss of my computer equipment. I don’t store the archives in the cloud, as I’ve got large databases and the upload times would be long, and the file sizes would count against my monthly usage limits. Instead, the archives are periodically updated to a portable drive stored in my bank safety deposit box. If needed, I could retrieve the archives much more quickly than if they were stored in the cloud.

I’m planning to do a “manual” off-site storage, unless I can improve performance for the cloud I think.

I’m not sure why it’s telling me it will take 12 hours to transfer my 1.8 GB Devon zip archive to my Dropbox. My 1.16 GB Scrivener backup took well less than an hour. Hmm.

I note that the Scrivener one is formatted as a bunch of individual files. I’m not sure how the large compressed zip file effects uploading.

I did note over at [url]archive zipped backup to DropBox failed because "file in use] that I think the long compression process may have given Dropbox a bit of grief when I used the archive script.

For now it looks like physical backups stored locally and off-site may be the way to go.

I am mystified why my Scrivener backups seem to work though.

What is actually the benefit of using the script vs the File > Export > Database Archive functionality?
Except the fact that the script does not ask for a location …

They are functionally the same, but…

  • The export command allows you to manually specify the name and location.

  • The script creates a datestamped ZIP file in a specific location, so there’s no need to manually name or choose the location.

1 Like

Thanks. Is there any way to have the script automatically be run at certain intervals?

You can create a smart rules for this.

Well, that’s responding to a posting I made 7 years ago. Long out of date. I’ve deleted that post now so as not to confuse more people.

I use Mac automator to move the file from Backup to a Onedrive folder at night so that the Onedrive upload does not impact on machine performance. We live at the end of a very long piece of copper wire so uploads can take what seems like a decade or two.You could use automator ( free and included in Mac OS) to move or copy your files to any location that suits.

Time Machine hourly.

BackBlaze offsite backup “continuously.”

SuperDuper full disk copy monthly to a rotating library of disks.

Active files databases synced via Dropbox. Yes, I understand that’s not a comprehensive backup strategy, but it will let me work on my iPad if my computer goes sideways.

You should be very cautious with this if DEVONthink databases are involved. Snapshot-style backups are considered safer. Also, there was recent discussion of a loss of metadata when usng BackBlaze.

Thanks for the heads up. Since both Time Machine and BackBlaze are running continuously, I’m not overly concerned about the problem of backing up an active database: both are active during many hours when I’m not using DT.

I’m willing to accept the metadata risk in exchange for the convenience. I don’t use metadata extensively, and BackBlaze is already the “backup of last resort” in that I won’t need it unless multiple on-site backups are corrupted or destroyed.

2 Likes

Just chiming in here to say that the continous option in Backblaze is actually every few hours. It usually takes the client software at least 2 hours (sometimes 4) to notice changes. I’m running Time Machine, Arq and Backblaze and just like you, I consider Backblaze as a last resort after TM and Arq and mostly use it to back up media files from an external drive that would be too expensive to back up with Arq.