I'm drowning in orphaned files

I rely on Backblaze for everything on my hard drive, not just DT and I’m grateful to have it work continuously, especially when I travel. Time Machine is wonderful, and I use it at the end of every day, but it requires a dedicated HD. Yes, SSDs have miniaturized, so could take one with me when I travel, but I have to plug it in and tell Time Machine to back up – not always convenient when a flight is announced, or whatever.

So if DT shouldn’t be backed up to either iCloud, or Backblaze, what then? Carry yet another SSD just for DT?

Or: Is it safe to continuously backup DT databases to Backblaze if everything is Indexed? – In other words, it’d be backing up all the indexed files on my hard drive separately, and all that’s in the DT databases are the aliases. Safe?

If not, which cloud apps are safe for DT?

See the chapter of the DEVONthink Manual I mentioned above. No need to repeat.

Look particularly at the archive files as a way for you to get good backups no matter where you put them.

1 Like

We have plenty of Arq users, including our company president.

Or: Is it safe to continuously backup DT databases to Backblaze if everything is Indexed? – In other words, it’d be backing up all the indexed files on my hard drive separately, and all that’s in the DT databases are the aliases. Safe?

No, I wouldn’t do that either. And there aren’t a bunch of aliases in the database. There’s much more going on than that.

1 Like

Hi Jim:

Thought I had a good system in place, but … expect the unexpected, right?

I chose Backblaze because it’s a backup program + unlimited storage with no expiration, it seemed secure, and isn’t expensive. Set it and forget it. The UI for restore is a bit clunky, but it works, and I haven’t had to use it very often.

But now I’ve been warned it can damage DT files. So while I figure out my options I excluded all DT files from Backblaze, and set it to On Demand instead of Continuous.

I can let Backblaze deal with all my other files, and for redundancy and travel, use Time Machine and a xTB external SSD, and never ever ever send a DT file up to the cloud. But that’s not as convenient as set it and forget it.

Did I read somewhere that it’s safe to backup a DT Archive file to Backblaze? If I save my DT files as .zip to another folder on my internal hard drive, can I let Backblaze back them up?

If it safe to put a DT .zip file in iCloud?

Or?

I get that you’re saying Arq can safely backup DT files, but it doesn’t provide storage. I avoid Google anything, had a bad experience with Dropbox and don’t need sharing capacity anyway, and not sure I want any more involvement with Amazon than I already have for other stuff. Alternative?

Only other backup service that also provides storage and gets good reviews is 1Drive, and it floats to the top of all listed reviews. Is it safe for DT files?

Really appreciate all your help with this – seeing scrambled DT files is very nervous-making!

June

Yes, the .zip from a database archive is safe to put wherever you want. However, it’s also a full backup not an incremental. So as your database grows, the time it takes to do a database archive increases. This is why doing this on a less frequent basis is recommended. In many corporate or academic settings, incrementals are done during the week and a full backup on the weekend.

PS: I can’t prove BackBlaze as the specific cause of your issues. However, DEVONthink databases are not just package files but ones with many moving parts in them. It’s not a mere container of files but also resources that are being changed and modified and open in memory as you work.

And while closing databases before a backup is ideal, the snapshot style backup “takes a picture” of your files and preserves the state of things at that time. This works even if databases are open.

3 Likes

It’s not the backup program itself – it’s the fact that it runs while the files are open.

I use Backblaze with Arq. Arq runs a script that closes DT before the backup and opens it again after the backup. That should avoid this kind of problem.

Yes to both, of course. Or rather: As long as the zip file is not being created while the backup runs.

It does.

2 Likes

Hi Christian, I’m interested in doing that - can you tell a little more how you use Backblaze (which I have) with Arq (which I would buy)?

Thanks!

[EDIT] - I found this, I guess you are using B2…

Probably. I set this up a long time ago and forgot about the details.

Arq allows you to define different storage locations (WebDAV server, Backblaze, Arq itself) for your backups, each with their own backup plans (time, files to backup, exclusions, etc.).

And for each backup plan, you can specify a script that runs before and one that runs after backup:

The first script:

#!/bin/zsh
for i in DNtp com.moneymoney-app.retail; do
/Users/ck/bin/startorstop.sh $i stop
done

And startorstop.sh:

#!/bin/zsh

msg="Usage: $0 applicationID start|stop."
if [[ $#  == 0 ]]; then
   echo "$msg"
   exit 1
fi

if [[ $2 = "start" ]];  then
  echo "Starting $1"
  osascript -l AppleScript -e "tell Application id \"$1\" to activate"
  exit 0
elif [[ $2 = "stop" ]]; then
  echo "Stopping $1"
  osascript -l AppleScript -e "tell Application id \"$1\" to if it is running then quit saving \"yes\" "
  exit 0
else
  echo "Wrong input parameter '$2'', use 'start' or 'stop'."
  exit 1
fi

Cool, thanks Christian!

So these are shell scripts, which I’ve never ever done, but happy to take as homework :slight_smile:

I suppose this is an array, correct?

for i in DNtp com.moneymoney-app.retail;

so if I have several DT databases, I would have to do something like

{com.foo; com.bar; com.meh}

is that correct? and startstop.sh would just iterate through the array?

/mike

Not in shell lingo – I suppose it’s a list there.

No, that’s a misconceptiong. “DNtp” and “moneymoney-app.retail” are the application IDs used the the startorstop.sh script. Basically the name of the applications to start or stop.

No. The start.sh and stop.sh scripts iterate through the list with for i in.
startstop.sh just runs AppleScript code to start and stop an application using if it is running then quit saving to stop and activate to start it.

It just looks ugly, but it’s quite straight forward (imo).

1 Like

thanks again Christian - I will be studying your script, and for sure will get Arq.

right now I have CCC, which has always been great and reliable - except - it does not backup to cloud (there is a hack, but I don’t want to use that).

I’ll keep CCC for the local backups.

How about directing CCC output to a USB drive, then let BackBlaze back that up? May not need Arc?

Ha ha! I’m doing that too.

Summary:

  1. TimeMachine (to a Thunderbolt 4 enclosure with an 8 TB hard drive)
  2. CCC of files on the Mac to a few 4 TB SSD - keeping at least 2 copies of everything, including Adobe Lightroom files and catalogs, DT databases.
  3. Backblaze classic of the Mac (sans system, apps, so it’s mainly the Documents folder and Desktop), and of an 8 TB hard drive that contains tons of legacy data, apps, music, photos - which I hope never to look at again but am too cowardly to get rid of :wink:

Christian’s Arq solution seemed very neat - but yeas, I hear you.

Thanks to all for the help you’ve offered. Have to say that one of the many wonderful things about DT is Support and this Forum!

Missed the part about Arq offering it’s own servers for the Premium version. I’ve now signed up for the trial and the first thing I noticed was how nice the UI is! Makes Backblaze look like an antique. Also impressed by how versatile it is as to when, where, and which folders get backed up.

Right now I have Arq set to backup to their servers only DT and files related to my WIP mss. Everything else is still being backed up to Backblaze. But why should I continue to use Backblaze at extra cost, when Arq offers its servers?

BTW: I have several DT databases for different aspects of life/work, so none of them is so big that saving an Archive zip is very time-consuming. And a plus: having several meant that my recent misadventure with corrupted backups meant that not all of them were damaged! Did it for my own convenience, and now realize it was risk avoidant.

2 Likes

Glad to help get things squared away and indeed we are fortunate to have a strong, knowledgeable, and vibrant community here. :heart: :slight_smile:

1 Like