Backing up DevonThink pro office database

DEVONthink Tips
1

Hi,
I am new to Apple Macs and DevonThink. I have just purchased DTPO to go paperless and excited about it. Today I spent 4 hours importing PDFs and scanning etc.
I am really paranoid about data loss and so I want to make sure I backup frequently.
I have read a few posts where Time Machine is recommended. I have a TM backup and it is about 300GB in size as I have lots of other data. I am then paranoid about anything happening to my external drive. Yes I can maybe get another drive and keep another TM copy etc.
To look after 300GB is not exactly portable. What I am looking to do is backup only the DTPO database etc (which will not be big) so I can store multiple backups in the cloud or to different media etc. Currently my DTPO (.dtbase2) file size is only 500MB.
I have seen some posts where people are suggesting ways to backup which seems scary to me as being a newbie to Mac and DTPO i am still finding my way round.
I thought that I could just backup everything by selecting an option from software and point to location. In the TOOLS menu there is ā€˜backup and optimiseā€™. What does this do? I am confused! Please help as all I want to be sure and feel confident that my data is safe with a simple backup procedure.
Is it just a matter of copying DatabaseName.dtBase2 file or is there more to it? If I decide to use a third party software, what would I need t backup.
I also wanted to install DTPO onto another Mac (different location) and transport the database to that machine as a disaster fallback and maybe I can test backup and recovery procedure.
Sorry for many questions and I hope someone can guide me (step by step idiot mode)

Thank you

2

I applaud you for having a backup system for all your data. Time Machine will protect you from a boot drive crash and other possible problems.

A good way to make additional backups of your DEVONthink Pro databases would be as Database Archives. These are the smallest possible zipped copies of your databases. Thereā€™s a menubar command, File > Export > Database Archive. The Scripts menu of DEVONthink Pro and Pro Office also has such a command: Scripts > Export > Backup Archive, which can save an archived database to a location of your choice, including an external drive, Dropbox, etc.

3

Hi. Here is my backup system. It has been working well for me over the last few years.
christopher-mayo.com/?p=962

It is pretty simple. Every morning I plug in the time machine drive. Every month I save my active stuff (including DT) over to my main backup drive. I replicate these drives so that I have everything in multiple physical locations. Itā€™s always possible to have data loss or data corruption, but I think the system I describe there makes it pretty unlikely.

The only change I have made over the last year is to stop using SpiderOak. It is a wonderful service, and I highly recommend it, but when I started a new position last spring it became redundant for my use case.

4

Thank you all for responding and sharing your thoughts. Much appreciated! :smiley:

Bill, Thank you for the FILE|EXPORT|DATABASE ARCHIVE suggestion. Can I name these archive snapshots like MyDataBase.dtBase2-1.zip ,MyDataBase.dtBase2-2.zip ,MyDataBase.dtBase2-3.zip etc or will it get upset?
I have taken an archive snapshot now and it asked for which extension and I chose ā€˜bothā€™.
What is the procedure if I need to recover or take an archive and import it into DTPO on another Mac?
Sorryā€¦more questions :confused:

5

This looks useful Bill. Another day, another something learnt about DTPO that I didnā€™t know was there.

I ran the script. It took long. I did something else. When I came back, everything was the way it was before. I have no idea where it saved that zip file, since I wasnā€™t asked. Any ideas??

6

@ korm: I agree that itā€™s a good idea to remember that all drives will eventually fail. For that reason, I change out the drives (I rotate two drives, one of which is stored off-site and switched every few days) that I use for Time Machine backups after about two years of use, and always make sure that the replacement drive is a high quality drive from a reputable manufacturer, and has considerably more storage space than the drive it is backing up.

The last two times Iā€™ve bought a new computer, Iā€™ve transferred data from the older computer to it via a Time Machine backup and had no problems. (As Iā€™ve been able to use a Thunderbolt connection from the backup drive, the transfer speed has been fast.)

@ Calapso & Cassady: Both the commands File > Export > Database Archive and Scripts > Export > Database Archive allow the user to select the location to which the Database Archive file will be saved. But the ā€œdefaultā€ location is already set, so make certain you check it and choose another location, such as a mounted external drive, before hitting OK!

The resulting Database Archive file for a database will automatically be given the Name of the database file plus the date the archive was created, plus the .zip filetype of the compressed database file.

To use a Database Archive, e.g., for recovery of a damaged database: This should be done with the DEVONthink application Quit. First double-click on the archive file to unzip it. Then move the uncompressed database file to replace the damaged file, so that it is in the location expected by DEVONthink. (Also, if recovering from Time Machine or other external backup system, do so while DEVONthink has been Quit.)

Regardless of the backup system used (except for Database Archive), the backup system will happily backup up DEVONthink databases that have errors. For that reason, I periodically run Tools > Verify & Repair on my databases (and immediately do so if anything flaky has happened). I havenā€™t seen database errors in a long time; my databases stay stable. But I stick with those habits!

A cautionary note: The older a backup file, the more likely loss of files added after the time of backup becomes. Thatā€™s one of the reasons I like Time Machine, as it is set to make hourly databases and will keep those for up to 24 hours. It then converts a set of hourly backups to daily backup, then weekly, etc.

So to avoid loss of work, if it is necessary to resort to an external backup file, choose the most recent one that will restore a working database with minimal loss of new content or edits made after that backup file was created.

With that in mind, if the database damage had occurred within the time span in which internal Backups had been made in the database, the first thing I might try would be use of DEVONthinkā€™s Tools > Restore Backup procedure. By default, there are three rotating internal backups of the state of the database. Iā€™ve got Preferences > Backup set for daily internal backups. So even if I had to resort to the third most recent internal Backup folder to return to a good working state of the database, any new files added since that backup was made wonā€™t be lost. The new files added to the Files.noindex folder inside the database will be found by running Verify & Repair and placed in an Orphan files group, from which they can be properly filed into the database. And of course any edits that had been made in the files contained in Files.noindex will also be up to date.

The next step I might take, if Restore Backup didnā€™t result in a working database, might be Tools > Rebuild Database, for the same reasons, to preserve contents of newly added or edited filesā€”especially if a lot of work on the database had been done subsequent to the time of the most recent working (i.e., ā€œgoodā€) external backup file.

There are preventive maintenance procedures that can help prevent database problems or correct them before damage escalates. 1) Verify & Repair to confirm that no database errors exist, or to become aware of them and correct them. 2) Appleā€™s Disk Utility app to check for and correct permissions errors and to check for Disk errors and correct them if found. 3) OS X maintenance procedures to keep the operating system clean, lean and efficient, either using Terminal.app procedures or (in a more user-friendly way) a utility such as C0cktail or OnyX (thatā€™s updated for the installed version of OS X).

1 Like
7

@korm Many thanks, found it.

@Bill ā€“ ta ā€“ sage advice, must remember to move it later, donā€™t want to burn through the 10GB data required to move it over to the cloud, just yet! 8)

8

Interesting discussion above.

On a related note, what are the panelā€™s views on letting back-ups run on open databases?

My practice is to run Time Machine as normal and a separate system of regular, scheduled SuperDuper back-ups but to exclude actual databases from the back-up process, in case the backup routine runs at the same time as me working on a database.

I am sure that I read that there is the potential to corrupt a database if a back-up routine runs whilst itā€™s open and included in the back-up routine. I apply the same thinking to all databases that I use (e.g. Lightroom or Aperture, which are photographic databases).

I do back-up the back-ups though - I independently back-up databases (for DTPO I use the archive scheme discussed above) to an external HDD and include those in my Time Machine and SuperDuper back-ups.

Thoughts?

9

There should be no worry of database corruption from TimeMachineā€™s mechanism. Itā€™s essentially taking a snapshot at a given point in time, say at 12:00 noon. Deltas from the previous backup to noon are cached (the ā€œPreparing Backupā€ message) as it determines what changes to Sync and what others will merely be hard links. Any changes to your database at 12:01 are ignored until the next backup preparation starts.

10

Thanks Jim.

11

In principle, there is one potential issue with Time Machine backups of open DEVONthink databases. At the moment the Time Machine ā€œsnapshotā€ of the database is taken, itā€™s possible that the database might have data in memory that hasnā€™t yet been saved to disk, and so will not be included in the backup. Backup of closed databases wouldnā€™t present that issue.

As a practical matter, I havenā€™t experienced problems with Time Machine backups related to that issue. As Iā€™ve set Time Machine to make incremental backups hourly, all the incremental changes saved to disk since the last backup run are incorporated, leaving only the potential of some yet unsaved data at the time of the new backup. The effect is that any data missing from the data on disk when the previous backup was made is ā€œpicked upā€ at the next backup. So even when a backup is made automatically at a time when Iā€™m making changes in the database, raising the potential of some data not yet having been saved to disk, the next backup will be made on current state of the data on disk, and so will include all data saved to disk since the previous backup was made.

Time Machine makes hourly backups for up to 24 hours, then sets that ā€œsetā€ of backups to a ā€˜dayā€™ period backup, then aggregates older daily backups, etc.

So while in theory it would be best to have databases closed each time Time Machine makes a backup, Iā€™m persuaded that the frequency with which databases are made and subsequently aggregated both helps mitigate the potential of data lost when databases are open at time of backup, and adds the advantage of return to a state of the database at a time ā€œjust beforeā€ Something Bad happened to the database.

A couple of times over the past several years Iā€™ve tested Time Machine backups of the same ā€œvintageā€ as backups made by the Database Archive procedure, and found no detectable differences. Same number of documents, etc. In principle there could be differences, in practice the potential for differences doesnā€™t seem large.

Given the potential of data loss when databases are open at time of backup, what I like about Time Machine is that it provides more backups (over the last 24 hours) than would be done if I used a backup system that functioned only daily on closed databases (or, worse, less frequent backups), and it does so automatically, without my intervention.

My two-part sermon:

  1. If you prefer a different backup system than Time Machine, fine. Use it. But do use a good backup system that allows retrieval of your data if Something Bad has happened.

  2. Because backup utilities will happily backup the state of a database even if itā€™s badly corrupted, develop the habit of periodically running Tools > Verify & Repair every few days, weekly or whatever works for you. If errors are found, use the tools in DEVONthink to eliminate them, or revert to a backup of the database if thatā€™s not possible.

12

Bill brings up a good point on what I affectionately call the ā€œshadow periodā€ for backups. These are times when an operator continues to work as a backup is being prepared and the current changes are essentially hidden to the backup. So if your backup fires off at noon and the next is an hour later, your changes wonā€™t be backed up for an hour even if the backup process is still running. So if your machine crashes at 12:30, your past half-hourā€™s work may be lost. This often confuses Users when they donā€™t find the files from this time backed up.

Also, thanks for bringing this up again, Bill as I donā€™t think we can stress this enoughā€¦

13

Another comment re Jimā€™s ā€œshadow periodā€: Once in a while Iā€™ll go wild and start doing a lot of reorganization of a database, creating new groups, perhaps in hierarchies, and moving lots of documents to new locations.

When I take a break, Iā€™ll run Tools > Verify & Repair to ensure there are no errors. If so, Iā€™ll then run Tools > Backup & Optimize to update the most recent internal Backup folder.

The effect is to make it easier to return to a good state of the database if, when I resume work on the database, I next do something stupid and wish to return to the prior state. Tools > Restore Backup can quickly revert to a state before my stupidity took hold.

Iā€™ll then manually initiate a Time Machine backup to make certain the most recent Time Machine backup is of an acceptable state of the database. By this time, all data that was in dynamic memory will have been saved to disk. Iā€™ve minimized the odds that, were I to need to restore from Time Machine, the restored database would contain my mistaken changesā€“and over time, Time Machine will eventually drop any restorable record of my mistakes.

If only it were so easy to correct other mistakes Iā€™ve made outside the DEVONthink environment! Iā€™ll never forget the mistake I made by ordering a supposedly Cajun dish at a supposedly Cajun restaurant in Portland, Oregon. I suffered a night of misery, without any opportunity to jump back in time and order a hamburger, instead. :slight_smile:

14

Hi

Thank you all for your feedback and suggestions.

Today I had a BIG scare. Let me explain:

I have noticed that since installing and running DTPO, I have had system crashes on my MacBook Pro. I normally (when I have finished using my MBP, I close the lid and put it away without shutting down. Since DTPO, when I have done the same I have noticed that my MBP does not wake up when the lid id opened. It is dead! I have to press the on/OFF key a few times to boot up. Has anyone had this issue?

Today I decided to close my database to avoid any corruption. I did this by FILE-Close Database Option. When I came to use it again my database was nowhere to be found. There was nothing in FILE-OPEN RECENT and when I tried FILE-OPEN i could not see my database anywhere. I searched for my database and no reference of it anywhere. Even un-hiding files did not show up my database. Only thing I saw was the ZIP files I had done via the FILE-EXPORT-DATABASE ARCHIVE.
I extracted the last archive and got my database back but had lost 2 hours work that I had scanned in. This is very worrying :cry:

I donā€™t feel comfortable with DTPO after this experience. Especially after being paranoid enough to start this thread. I feel that if this software is going to be my repository of all my valuable and important data, then the backup restore thing needs to be addressed clearly for all especially a newbie like me. All database applications I have used in my time have had a simple procedure to backup. Better still most of these do a quick archive every time you exit the application with the date & time stamp on the zip file. You could then set a limit of say keep a cycle of 20 zip files etc which can then be copied manually to external or cloud storage as you see fit.
How come DTPO does not adopt a simple procedure like this? Instead we have got:

FILE-EXPORT-DATABASE ARCHIVE
TOOLS- VERIFY & REPAIR
TOOLS- BACKUP & OPTIMISE
TOOLS- RESTORE DATABASE
TOOLS- REBUILD DATABASE

I find all this confusing and a bit misleading. I originally thought that TOOLS- BACKUP & OPTIMISE did everything for you but I am not sure what this does. I am fast loosing confidence with this software and wondering whether to return it.

Sorry i sound negative but I am short on confidence after my experience today.

15

Hi. I canā€™t comment on your experience, because I havenā€™t seen anything like it, but I would strongly recommend using TimeMachine (see comments in previous posts). It isnā€™t a magic bullet. It is a very powerful and easy-to-use tool that will pretty much accomplish what you are wanting (backup with versions), but it does it with everything on your local drive.

16

@Calapso

The problems with your machine (and the software) are NOT things that a user forum can resolve. I strongly suggest you open a Support ticket with DEVONtech, and call Apple support or make an appointment at the Genius bar. If your machine is failing and/or the software is corrupt and youā€™re losing your data, then you need professional one-one support.

17

I second kormā€™s suggestion especially that of contacting Apple through their Genius Bar, I had a similar problem a long time ago and the guy at Apple solved it straight away, I can not remember how but it was done there and then. Also might be worth having a look at Appleā€™s support community online. I would be most surprised if it has anything to do with DEVONthink.

18

Thank you korm.

I have opened a support ticket and see what happens. I also contacted Apple and they say it is more than likely to do with Devonthink! So I am none the wiser. Hope the guys at DT can offer some suggestions

19

Sorry for commenting a little bit harsh: Did you ever read the manual (which is really detailed)?

Iā€™m working with DTPO for years with tens of thousands of documents without having your experience for even one timeā€¦

20

No need to be harsh and yes i did read the manual. As I explained on my post that I am a newbie both on the Mac world and DevonThink so I am a bit nervous and paranoid so I want to make sure I ask the dumb questions to get some reassurance. Last thing I want is thinking I am doing ok and then loose data.

You are obviously a more experinced user with tens of thousands of documents so maybe you would like to share how you manage your backups and manage your database/s etc which would be more helpful and productive for me and maybe others. BTW how big are your database/s? I know itā€™s slightly off topic but just curious to know you use DTPO if I am not intruding.

Thank you