Are .dtmeta files required for restoring my DB?

DEVONthink Tips
1

In addition to using Time Machine, I’m doing incremental cloud backups using Arq. Do I need to backup my various (e.g. ~10) .dtmeta files in the data base package on a continuous basis? These are often quite large (e.g. gigabytes), and I’m wondering if there are redundant and would not be required if I needed to restore my database from the cloud (not my first choice, I do local backups as well and hopefully could use them)?

2

Yes. These files contain e.g. metadata, thumbnails, the group hierarchy and the search index. But Backup subfolders are not required.

3

Thanks very much. Unless I’m missing something there is no point excluding the backup subfolders, as they are already uploaded and the Arq docs say that pattern searching to exclude them would just slow down the backup further. I guess if someone was really concerned about the backing up these files they should use multiple data bases (I put everything into one).

4

While we’re discussing those, what about the .dtMeta~ files? Are those also backup files? If so, what is the difference between those and the ones in Backup/?

5

These files are probably orphans, what’s their creation/modification date?

6

They’re very recent, which is one reason I suspected they were backup files. (The other is the ~ suffix.)

  -rw-r--r--   1 ats  staff    1575923 Jul  6 19:11 DEVONthink-1.dtMeta
  -rw-r--r--   1 ats  staff    1456942 Jul  4 17:49 DEVONthink-1.dtMeta~
  -rw-r--r--   1 ats  staff      90175 Jul  6 19:11 DEVONthink-10.dtMeta
  -rw-r--r--   1 ats  staff      81319 Jun 20 10:46 DEVONthink-10.dtMeta~
  -rw-r--r--   1 ats  staff     234095 Jul  6 19:11 DEVONthink-2.dtMeta
  -rw-r--r--   1 ats  staff     281368 Jun 20 10:46 DEVONthink-2.dtMeta~
  -rw-r--r--   1 ats  staff       3312 Jun  2 09:31 DEVONthink-3.dtMeta
  -rw-r--r--   1 ats  staff       3312 Jun  2 09:31 DEVONthink-3.dtMeta~
  -rw-r--r--   1 ats  staff     258070 Jul  6 19:11 DEVONthink-4.dtMeta
  -rw-r--r--   1 ats  staff     347035 Jun  2 09:31 DEVONthink-4.dtMeta~
  -rw-r--r--   1 ats  staff    8967118 Jul  6 19:11 DEVONthink-5.dtMeta
  -rw-r--r--   1 ats  staff    8572783 Jun 20 10:46 DEVONthink-5.dtMeta~
  -rw-r--r--   1 ats  staff         22 Jun  2 09:31 DEVONthink-6.dtMeta
  -rw-r--r--   1 ats  staff         22 Jun  2 09:31 DEVONthink-6.dtMeta~
  -rw-r--r--   1 ats  staff      49912 Jul  6 19:11 DEVONthink-7.dtMeta
  -rw-r--r--   1 ats  staff      43184 Jun 20 10:46 DEVONthink-7.dtMeta~
  -rw-r--r--   1 ats  staff    6112172 Jul  6 19:11 DEVONthink-8.dtMeta
  -rw-r--r--   1 ats  staff    5574464 Jun 20 10:46 DEVONthink-8.dtMeta~
  -rw-r--r--   1 ats  staff  101849356 Jul  6 19:11 DEVONthink-9.dtMeta
  -rw-r--r--   1 ats  staff   94396338 Jun 20 10:46 DEVONthink-9.dtMeta~
7

Did DEVONthink crash lately or did you use “Force quit”?

8

Hard reboot most likely. I’ve been testing Yosemite on some of my machines, and there have been a few times I’ve kernel panicked or had to hard reboot.

9

This might indeed have caused these orphans.

10

Good to know. I’ll feel no qualms about deleting them whenever I see them.

11

@rjs: By default, a database will contain three internal Backup folders that are sequentially updated by user action (Tools > Backup & Optimize or on the schedule set in Preferences > Backup.

Those internal Backup folders contain indexing and metadata about the files captured to the database. They will grow in size as the database grows.

The smallest possible backup of a database is a database archive, which is a zipped copy of the database that contains no internal Backup folder. To restore from a database archive, double-click to unzip it and open the resulting database. In operation, the internal Backup folders will be recreated.

Database archives can be created via File > Export > Database Archive or by a script, e.g., Scripts > Export > Database Archive (or Daily Backup Archive).

Most of the time required to create a database archive is the (automatic) final step to compress (Zip) the database backup file. Compression usually dramatically reduces the file size of the database backup for storage locally or in the cloud.

When the DEVONthink procedure is run to create a database archive the procedure will first check for errors, and will stop and report that errors exist. The user should then take action to correct problems, e.g., by running Tools > Rebuild Database. That’s great, as backups that contain errors may be less than optimally useful. (After running Rebuild Database, check the Log (Window > Log) for a list of files that may not have been included in the rebuilt database; the list can be saved for future reference. Then run Tools > Verify & Repair again and refile any documents that may appear in an Orphans group.)

Always remember that other backup procedures (e.g., Time Machine, ChronoSync) do not check automatically for database errors. So it’s a good idea to manually run Tools > Verify & Repair at least every week or so of database use, and always after seeing any flaky behavior or after a Force Quit or other event that doesn’t properly close open databases.