Large database backups ALL failing after beta 6

DEVONthink Tips
1

I have a problem that did not exist before updating to devonthink pro office beta 6, I ran 1 through 5 with no issues. Since updating to 6 I am no longer able to run devonthink backups.

To be more precise, the script menu commands for export Daily Backup, have begun failing, as has the new Files > Export > backup command.

The part where it always fails now, no matter what database I am using, is when it attempts to copy over the DT-metadata files. All the contents of the Files.noindex folders appears to be intact and have no problems. I have also tried restoring backups using Time Machine and clones; these also have no problems at all.

Did something strange happen after the OS/X 10.5.8 update, or is it Devonthink itself? I also Verify/Optimize then run Backup Database within DevonThink itself, and these also give me no problems, no checksum errors, no files lost errors, nothing.

Since all my databases seem to be running problem free, and since I have current backups, up until the moment this started happening, at the moment I am simply doing nothing. I almost suspect that something is wrong with the zip command itself, or how it is being called.

Yes I know I should reimport all the databases, but realistically here, there are some which contain 100,000+ files, and I am not going to sit there and watch the little blue line move all day long to do this with every database, because either Devonthink messed up some archiving command that used to work fine (and it STILL DOES, on small databases, those that are no more than 800MB continue to Zip with no problems, the databases that are over 1GB, have ALL begun failing since beta 6), or because OS/X 10.5.8 has done god knows what.

To be clear: all databases which produce zip file backups smaller than around 800MB continue working problem-free, somewhere around 900MB/over 1GB produces total fail, across the board, on all my databases (total fail defined, to repeat and be clear: the zip files are unzippable and seem to continue the full content of all my databses in the FIles.noindex folder. What’s not present in any of them, is the Dt-metadata files), since beta 6, combined with 10.5.8 updates.

2

I know it’s frustrating when I reply that I can’t replicate that problem here. :slight_smile:

Running DT Pro Office 2.0 public beta 6 under OS X 10.5.8, I just ran Daily Backup Archive on one of my databases. The backup procedure ran to completion.

The file size of the database is 3.65 GB on disk. The file size of the zipped archive is 1.74 GB.

I then unzipped the archived database and confirmed that it held the same content as the original.

[Scratching head] Is it possible that you have a third-party extension of some kind that may be causing errors after the update to OS X 10.5.8? If so, that might be causing flaky behavior. There are a number of utilities that ‘hack’ code in the operating system – Safari add-ons, multiple clipboards, preference panes, QuickTIme media extensions, etc. If you run DTPO2 when logged in as Guest, does Daily Backup Archive still fail? (If you try that as a test, run Help > Install Add-ons but first uncheck the Mail plugin and Abby installations, as they would not be relevant to the test.) [/Scratching head]

Anyone else reporting success or failure in running Daily Backup Archive with pb6 and under OS X 10.5.8?

3

FAIL during the Verify stage:

Database is damaged.png
Log message:

13:59:06: Apple/Mac > VersionTracker > Comments > iPalette - rgb ≠ RGB RGB is correct! Missing File

But this file does exist in the database:

…/Files.noindex/php/0/iPalette - rgb ≠ RGB RGB is correct!.php

For some reason Tools > Verify & Repair (Shift-Command-Y) doesn’t find that error (oh-oh).

Problem is with the ≠ character encoding. I can reproduce it simply by importing the same file into DTPO again, creating a document item in the database that can’t locate the file.

4

Something very weird is happening… My main database just went on a diet and when I backup and optimize, it has gone from 600MB, to 512… Same database, with perhaps 20 new files added to it.

This has happened between beta 5 and beta 6. Where did 88+ MB of Dt-metadata index files just go? When I check the file size of Files.noindex, the collective content of those dirs has grown by exactly 1MB, everything appears to be alright in there, I can’t find anything I am missing in particular but something is very wonky with the indexing.

5

That’s nothing to get alarmed about. The database file “sprawls” a bit under use and when new content is occupied, taking up more disk space. The Tools > Backup & Optimize routine ‘compacts’ the database, often reducing the package file size on disk.

6

How did you notice this exact point of failing? All I get is the message “Backup failed”.

In Konsole I found this message today:

“08.08.09 11:36:33 DEVONthink Pro[536] Custom ColorPicker class with name .com.freeverselib could not be loaded.”

7
8

I looked at the .zip file that failed, attempted to uncompress it, succeeded, there was no system message that the file is corrupt when unpacking the .zip from Finder. Upon opening it, I have what seem to be all the correct files in Files.noindex (I am not really checking file by file, I am just using the df command from a terminal) but there is nothing at all in the top level directory or anywhere else, where there would normally be several Devonthink .dtMeta files, numbered 1-9. There are no files present. That’s what zip appears to be failing to compress and thus the message you get, which is the same one I get.

In other words the only thing the failed zip archive contains is a folder called Files.noindex, which seems to be ok.

I can zip the database from Finder with no trouble. I will experiment with the guest account when I have more free time this weekend Bill, thanks for the suggestion. I do not have any input managers, Safari add ons, I do have Quicktime codecs installed for Perian and Flip4Mac but these are not exactly obscure or weird codecs, I would guess half or more of all Mac users have them (they allow display of flash and .wmv files among others, using Quicktime).

If beta 6 is no longer able to find and index files with various character names in them, this would possibly explain the weight loss of the indexes and some other issues.

9

I tried to make the Daily Backup Archive with a database of 2,5 GB. No errors, zipped as usual, the zipped version now is 1,1 GB. I tried to unzip, got an error message:

DT Error entpacken .png

I unzipped without problems with Stuffit Expander, the database seems to be okay, including Metadata, Files and Settings, 2,15 GB, same size as the original database without internal backups in Finder.

Dealing with the above mentioned error message I renamed the Backup Folder to ‚DT Archive‘, and I’ve been able to unzip one of the failed Backups with Apples Archive Utility without any error message. (I suppose there are some inconsistencies for those like me, who have Apple’s Backup.app installed.)

Regarding the failed Backups: I can confirm, what you wrote in your last posting: There are the files but no Metadata.

10

Have you installed Lineform at some point? Anyway, whether you have or not, there is a file .com.freeverselib in /Library/Colorpickers/ (or ~/Library/Colorpickers/, I don’t remember). I found it and erased it on my system.

Not too sure what it has to do with DT, but in my case it kept logging this error to the console with many programs. So whoever decided to put it there (freeverse, presumably) not only are dishonest (unless they simply forgot to tell me what they were doing to my computer) but are also quite obviously incompetent, too.

11

The folder ColorPickers is empty. I don’t have installed Lineform. But I have Periscope.app installed, inside this application I found in the Contents folder FreeverseLib.framework. What could it have to do with DevonThink? Should I delete Periscope?

12

I have no idea what it has to do with DT, but in my case this error got logged to the console randomly (ie by random programs having nothing to do with any freeverse program). I don’t know why, I mailed freeverse and they ignored me.

Are you sure that it’s not in colorpickers? its name is .com.freeverselib, so it’s normally invisible in the finder. you’d have to set the finder to show invisible files or just use the terminal to look.

At any rate, I would just leave it alone if I were you. I do not think it causes any problem. and if you do want to get rid of it, you have to delete .com.freeverselib from ~/Library/ColorPickers, not periscope itself (although I suppose that periscope will add it back when you start it).

13

Or use DTech’s own EasyFind. :slight_smile:

14

s.hoffman did you have time to try this? Or solved the problem?

I’m a little bit exhausted by zipping and unzipping and exporting and importing databases all day long. I don’t want to use DevonThink at this state, but it’s one of the three most important applications fo my work. So at the moment I got really stuck with my work and trust to my research.

15

@Ursula I did run the tests on the guest account, using the new File > Export > Database Archive command. Same problems, but more than usual because my database is a combination of imported and indexed files and the guest account does not have permissions for some of the indexed files. My root account is enabled, so just for the heck of it, I ran it as root to make sure there are no permission issues left (yes I know this is a terrible idea, but at this point I am nearing the end of my patience and plan to do a clean wipe reinstall, the minute snow leopard is released next month), results are the same.

Same in this case being: devonthink gives a message, “Zip Archive Failed” it produces a Zip of roughly the correct size (slightly larger than my last successful zip using beta 5, indicating I’ve added some content), I can unzip this file using Finder, it contains Files.noindex and everything appears to be intact, but there are no Dt-Meta anything files.

Like you Ursula, I am a little tired of all this, at this stage I am giving up on testing, I’m not a developer, I do not have the time to sit in front of the computer spending hours testing to see if the backup really backs up or fails, I am not going to invest another 20 hours of time to try this, that or the other thing. It all worked under beta 5, it no longer works under beta 6, I do not know what the problem might be, I can restore the current database using both clones (copies of the disk) and Time Machine. Neither method gives any errors or problems.

I hope the problems are resolved at some point, I am unsure what my current plans are since my trusted system is no longer trusted, balanced against never having had any problems with Devonthink before and yes I realize I am using beta software, so for the moment I have stopped adding content to devonthink, hope it will get resolved at some time in the future and if I have to somehow lift the entire mess out of Files.noindex by hand, that’s really too bad and an incredible waste of my time but at least I know it’s in there and seems intact.

Summarizing as clearly as possible: when the Zip archive fails, what it seems to fail on, are the Dt-metadata files. Everything else is contained within the zip archive, but there are no dt-metadata files present. None of the databases which fail to archive produce any other errors, I can run verify, optimize to my heart’s content, I can copy the databases using Finder, reload them, no problems.

I am annoyed at the waste of time but have calmed down a lot, since all my data seems to be intact.

16

One workaround until a new beta will be available are the following older scripts:


-- Export daily backup.
-- Created by Christian Grunenberg on Wed May 12 2004.
-- Copyright (c) 2004-2009. All rights reserved.

tell application id "com.devon-technologies.thinkpro2"
	try
		set this_database to current database
		set these_records to the records of this_database
		set this_date to do shell script "date +%y:%m:%d"
		
		set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
		set this_path to path of this_database
		set this_name to the last text item of this_path
		set this_path to "~/Backup/" & this_name & " Exported/" & this_date
		set AppleScript's text item delimiters to od
		
		repeat with this_record in these_records
			export record this_record to this_path
		end repeat
	on error error_message number error_number
		if the error_number is not -128 then display alert "DEVONthink Pro" message error_message as warning
	end try
end tell



-- Export Backup Archive.
-- Created by Christian Grunenberg on Fri Jul 22 2005.
-- Copyright (c) 2005-2009. All rights reserved.

tell application id "com.devon-technologies.thinkpro2"
	try
		if not (exists current database) then error "No database is open."
		set this_database to the current database
		
		set this_path to path of this_database
		set {od, AppleScript's text item delimiters} to {AppleScript's text item delimiters, "/"}
		set this_filename to text item -1 of this_path
		set AppleScript's text item delimiters to od
		
		set tmpDir to POSIX path of (path to temporary items from user domain)
		set tmpDir to tmpDir & "/DEVONthink Pro/" & this_filename
		set this_date to do shell script "date +%y-%m-%d"
		
		set theFile to choose file name default name (this_filename & " " & this_date & ".zip")
		set this_archive to POSIX path of theFile
		if this_archive does not end with ".zip" then set this_archive to this_archive & ".zip"
		
		show progress indicator "Preparing Backup" steps 5
		
		with timeout of 600 seconds
			step progress indicator "Verifying..."
			if (verify database this_database) is not 0 then error "Database is damaged."
			
			step progress indicator "Optimizing..."
			if not (optimize database this_database) then error "Optimization of database failed."
			
			step progress indicator "Copying..."
			if not (backup database this_database to tmpDir with including files) then error "Backup failed."
			
			try
				set theSettingsSrc to (path of this_database) & "/Settings.plist"
				set theSettingsDst to tmpDir & "/Settings.plist"
				do shell script "cp " & quoted form of theSettingsSrc & " " & quoted form of theSettingsDst
			end try
			
			step progress indicator "Zipping..."
			do shell script "/usr/bin/ditto -c -k -rsrc --keepParent " & (quoted form of tmpDir) & " " & (quoted form of this_archive)
			step progress indicator "Cleaning Up..."
			try
				do shell script "rm -fR " & (quoted form of tmpDir)
			end try
		end timeout
		
		hide progress indicator
	on error error_message number error_number
		hide progress indicator
		if the error_number is not -128 then display alert "DEVONthink Pro" message error_message as warning
	end try
end tell

Just save them for example in the folder ~/Library/Application Support/DEVONthink Pro 2/Scripts/Export.

17

Thank you, it’s very impressive how quickly both devon technology people and other customers interact and respond to problems in these forums. I am finding devonthink very useful and have gone back to using it after verify the full contents of current and previous databases to make sure they match. They do.

I appreciate the time you took to post the older scripts, I don’t need to zip anything urgently, I am sticking with the system I am using now that works for me which are rotating clones and Time Machine, both of which produce no errors with any of my databases. If I need to zip I can do it from the command line or Finder for now.