I’ve just restored a database from Time Machine, a big one with a lot of words. When I’ve opened it, DT alerted of 50 errors, after successfully repair it, now I have 43 replicants that aren’t. They are the real file and there is only one instance and no replicants. No indexed items, only imported. Two questions arise here:
Well, this is getting a little bit too much weird.
From 9 databases recovered from Time Machine, 3 were corrupt far away of the possibly of recovery. In all 3 cases, a Verify and Repair aparente did the job, but a DT close and open had new issues until I was not confident anymore of the integrity of them. Inconsistencies plus files not found, plus excess of files, plus the situation of above.
I restored the 3 databases from 3 different snapshots of Time Machine with same behaviour, then I can discard a specific Time Machine backup corruption.
I’ve recovered them from the sync database, and from that, for now, no errors happened, but my question still is here:
(I’m not really in risk of lose anything, as all imported data is duplicated out of DT in my NAS, Dropbox and an external 2TB disk, but this kind of issues makes me thing if is it still worth continue using macOS for serious tasks, as I’m sure the main problem is not DT, but macOS itself. Currently I have some file-id non-recoverable errors not related to DT, a format and reinstall solves the issue a couple of days and it reproduces, and I’m sure it is not DT related because I checked with DT completely removed and the disk errors persisted, and they aren’t physical as well but this new piece of sh*t that is APFS. A lot of screensavers but real and nasty bugs hidden below the carpet, “the best Mac we’ve done ever” (with extra epilepsy free of charge), “the best iPad we’ve donde ever” (with jellyfish effect), "the best mouse we’ve ever done (not usable while charging) and this evening in Europe, morning in the US, as an exclusive new: “the best iPhone we’ve done ever” (with no silence button but a push one, with no indication if the phone is in silence or not, and don’t forget the island in the non-pro phones, removing more screen space in exchange of a very stupid animation that can be done without island)).
That’s unfortunately not an unusual issue of Time Machine snapshots created while databases were in use at the same time. Using a different snapshot should usually help, or, if available, your own archived backups.
Anyway, does rebuilding fix the database?
Christian,I didn’t tried rebuild the database, I restored the offending ones from the sync database.
Now, knowing Time Machine is not valid for this, I will change the backup program this evening. I have a licences of CCC 6. Will it be more reliable?
Any solution taking snapshots of databases that are currently in use might experience such issues. E.g. a snapshot done while DEVONthink modifies one or more of its various database files & indexes and/or moves/renames/adds/deletes documents in its Files.noindex folder at the same time is most likely to have issues, e.g. orphaned/missing files and/or broken database structure. Repairing/rebuilding might fix these but no guarantee.
But of course there are usually lots of snapshots and therefore if one is broken, a different one should work. There’s also a series about backups on our blog:
What I do, and I think this a reliable backup method… I take a “Daily Archive Backup” which is a “zip” of the database using the script provided by DEVONthink. As other posts here have provided, I automate that with a scheduled job. TimeMachine, in addition to getting the DEVONthink database “packages” is also backing up those Zip files. I’m hoping/guessing that the Zip files are complete. So if my TimeMachine restore (if/when needed) doesn’t work, I have those Zip archives to fall back on.
Same. I think we’ve discussed Time Machine and “snapshot” backups before in the forum and it made me wary.
I do always make sure DT is closed when Time Machine is running (I run Time Machine manually). But after I’ve run Time Machine I take zip archives of all my databases on a different external hard-drive.
I restored from those zips earlier this year without issue so feel comfortable with this process.
Thanks, @cgrunenberg, I will read those entries later when at home.
@rmschne, @MsLogica, my databases use about half TB, and it is not reliable to me do a daily backup like that. What I do is zip and copy to an external disk once a month after doing a verification and closing DT. I could reserve some space in my 5TB backup disk, not Time Machine the Databases folder, and then use the daily script weekly, maintaining two copies. Delete the oldest one, newer remains, make a new one, and cycle weekly. I will think about this after I read the blog entries and will be here with my solution.
Thanks to all.
I’m assuming you use the Finder’s “Archive” command or some non-DEVONthink thing. I’m guessing that ok, but I use the DEVONthink “Daily Archive” script which also creates a Zip file and I assume it does whatever is needed to fully and reliably handle the open DEVONthink file. While labeled “Daily”, I don’t do it daily.
Well, I manually use Keka to generate 7zip archives, which compress more than bare ZIP, but yes, the right thing is using the script.
Well, after some messing the the daily backup script, and having adapted to backup all databases instead of the current active one, I’m doing it other way. I don’t need TM backing up each hour all my stuff, then I’ve set TM to manually backup and I’ve created a cron job to:
With this approach I kill two birds from one shot: periodically restart DT and having incremental safe TM backups.
1 1 * * * osascript -e 'quit app "DEVONthink 3"'
10 1 * * * tmutil startbackup
10 7 * * * open /Applications/DEVONthink\ 3.app/
At 1:01 in the morning, it stops DT, at 1:10 starts the backup, and at 7:10 starts DT. I’ve run the 3 commands separately to test they work and now they are in a crontab. This night we will see how this works.
BTW, this is my modified not used script (surely not the best way to loop over the open databases but it worked):
-- Daily backup archive.
-- Created by Christian Grunenberg on Mon Jun 22 2009.
-- Copyright (c) 2009-2022. All rights reserved.
tell application id "DNtp"
try
repeat with eachDatabase in every database
if not (exists current database) then error "No database is open."
set this_database to eachDatabase
set this_date to do shell script "date +%y-%m-%d"
set this_path to path of this_database
set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
set this_name to the last text item of this_path
set AppleScript's text item delimiters to ""
if this_name ends with ".dtBase2" then set this_name to (characters 1 thru -9 of this_name) as string
set this_archive to "/Volumes/Devonthink Backup/Backup" & this_name & " " & this_date & ".dtBase2.zip"
set AppleScript's text item delimiters to od
show progress indicator "Daily Backup Archive" steps 3
with timeout of 3600 seconds
step progress indicator "Verifying..."
if (verify database this_database) is not 0 then error "Database is damaged."
step progress indicator "Optimizing..."
if not (optimize database this_database) then error "Optimization of database failed."
step progress indicator "Zipping..."
if not (compress database this_database to this_archive) then error "Backup failed."
end timeout
hide progress indicator
end repeat
on error error_message number error_number
hide progress indicator
if the error_number is not -128 then display alert "DEVONthink" message error_message as warning
end try
end tell
This could be simplified too…
repeat with this_database in every database
Interesting. Thanks. You could add as new script, “All Open Databases Daily Backup” or similar title.
BTW, my cron jobs to stop, backup via TM, and start DT worked fine. In a couple of days I will recover one from TM and check it.