Dataloss in Think When Files Modified

Hi,

My DT1.8.1a works for me (under OS X 10.3.3) as long as I don’t modify any files to it; try to convert to the new database format (in the latter case, the process merely fails); or import the wrong type of file.

In the case in which I modify files in Think, they become corrupted if the application crashes, system loses power, etc., even if they have been saved, and are often not recovered after verify and compress.  This has happened to me approximately once each month - wiping out to-do lists, notes in preparation for a talk, etc.  At such times one feels like the technology has one undoing work, rather than making progress.

I doubt that I am the only individual experiencing this, but the various behaviours make the software almost unusable for any valuable information.  Rather than "save" regularly while typing, one must instead "backup database" regularly while typing!  the process reminds me of working with floppy disks in the early 80s.

As a registered user, I’m fed up.  Can I surrender my license for a refund?

Yon

The only time I’ve experienced data loss is after several computer crashes when DT was running. And that was when using a beta. I had just spent the weekend (literally 15 hours or so) importing and organizing my data. I was not happy when, on Monday I couldn’t access the database. However I had several hard crashes over the weekend and that’s what corrupted my data. It was not directly related to a DT bug or flaw.

Are you saying that you cannot modify any data without losing the entire database?

Have you performed maintenance on your computer? It sounds rather strange.

Also, do you have third party haxies or other applications/utilities running?(even something that runs in the background or in your menu bar etc) If so, disable them and try using DT again.

Try this before giving up:

I would back up my DT data (copy it to a CD or other removable media if you can). Delete the DT preferences (.plist files) and delete the DT app. In fact delete anything related to DEVON technologies, except your database and data files. (Make sure you back up any archives from DEVONagent etc)

I’d then boot from my Panther Install disk and run Disk Utility to repair permissions and run disk first aid.
Next I’d reinstall  the latest version of DT and create a new database by creating a few simple test files. Perhaps a PDF and ann rtf. Try to verify, modify etc your data. Does it function normally?

If it does, quit DT, drag the original DT database back into the library ->application support folder. Launch DT and try to modify, verify etc your DT data. Does it function normally?

If not, your data may be corrupt and you might try this to verify:

Create a new user account on your Mac and log in as that user.

Download a copy of DT and perform a fresh install of DT into that account. Try to create a few test documents in DT. If you can save/modify those without problems then try importing some of your original DT database files (This is why I suggest a CD or other removable media. Either way make sure that the DT folder you backed up initially is either on removable media or in a shared location otherwise you won’t be able to access it from both accounts).

If DT functions normally with some of those files in the new user account, import the balance of the database. If it still functions normally and the imported database allows you to modify the DT data without problems then there’s likely an issue/conflict in your original user account.

If it doesn’t function normally after an import of any original data, I’d say there’s a very good chance your database may be suspect and perhaps Christian can attempt a “fix”.

Hope this helps and good luck.

I’m saying I cannot reliably modify or create any important data, because there is a statistically important chance that the data will be lost, unless I go to unreasonable measures, and that empirically this chance is realized about once each month.

I appreciate your detailed suggestions, Milhouse.  I don’t know about other users, but I’m personally not interested in experimenting with trying to make this software stable on my computer (about which, nothing unusual is running in terms of system customizations, etc).  I am accustomed to do so with, for example, real-time audio software; If there are system-dependent issues, they can be quite hairy to sort out.

I’ve been through many and frequently-released versions of think in search of something stable enough for moderate use, without success, and I don’t see unique features which think offers as justifying continuing this effort.

My priority for the moment is carefully moving some valuable data I have presently in think to another location, and then deleting the application.

If Devon’s willing, I am also proposing the surrender of my registration number, in exchange for a refund or partial refund.

Yon

Sorry that you don’t appreciate DEVONthink, but IMHO your problem is not with the application but with recurring system/disk corruption on your computer.

I’ve been using DT since version 1.4. My database is comprised of thousands of documents and is currently over 700 MB in size. I’ve NEVER lost data, although I use DT every day (often running betas or alphas of upcoming versions), constantly adding and modifying content.

I set DT’s preferences to flush new data to disk every 3 minutes. I routinely run DT’s Verify & Repair tool, although I haven’t seen an error report in months, and the few errors detected have always been repaired. I make backup archives of my DEVONthink database, just in case. But I have not had to revert to a previous version, because the original database just keeps working. :slight_smile:

Mac OS X is wonderfully stable, but it DOES require routine housekeeping, or problems WILL occur. That’s not the fault of DEVONthink. I use C*cktail and DiskWarrior 3.01 as needed to keep my computer running quickly and stably.

Hi Bill,

I am a programmer, do know how to maintain my computer, and run maintenance programs as needed.  

In addition to the frustrating loss of files, my impression of devonthink is further colored by two things.  

First, despite heavy daily use and occasional hard crashes of the computer, I do not experience comparable data loss or other problems with any other application than Devon Think.

Second, I have had several other problems with Think, despite cleaning old versions and installing new or updated ones, and rebuilding the database on more than one occasion.  None of the crashes or other behaviours that I reported was explained.

Thanks,
Yon

Yon:

I didn’t mean to seem condescending, but most of the serious problems with DEVONthink happen to users who have System or disk corruption problems.

Perhaps my major criticism of OS X is that it does require user intervention to keep the operating system running smoothly. Many users don’t understand this, or don’t bother to do routine housekeeping on their Mac. The result can be a less than satisfactory user experience. I’m heartened by the rumors that the next one or two major upgrades of OS X will go a long way toward solving this problem.

An all too common problem results from power outages or brownouts, especially when data is being written to disk. I work on a PowerBook G4. The lights went out twice a couple of weeks ago, but I was able to keep on keeping on. For desktops, a UPS is advisable.

It’s always a good idea to make a backup of a database while everything is working properly. I do that fairly often, especially when I’ve been adding lots of items. Better yet, backups should be done on external media, in case the computer dies or is stolen! So far (knock on wood) I’ve never had a hard drive die on any of the Macs I’ve used over the past 20 years. My backups give me comfort, but I’ve never had to use them. The DT database just keeps working.

I had a hard crash yesterday, the third one since installing the original version of OS X and keeping current with subsequent releases and updates. I was running an alpha of DEVONthink Pro and doing scanning, scan touchups and OCR. What killed me was that I downloaded an update of the Flash plugin, which scrambled permissions. The next time I launched Safari, CRASH – with a message to hold down the power button to restart. I ran DiskWarrior 3.0.1, then a C*cktail suite including permissions repair.

On restart, everything ran fine until I launched Safari. It froze. I killed Safari and, on a hunch, looked at Safari’s preference file. It was much larger than normal, and had some really strange stuff in it. Deleted the file, relaunched Safari, and everything is fine.

My 700+ MB DT database is fine and just keeps ticking along. :slight_smile:

I’d also keep an eye on console and system logs for any suspicious activity:

/Library/Logs/Console/$USER/console.log
/var/log/system.log

Thanks for pointing that out. I agree – that’s highly recommended and I looked at them. But finding Safari’s corrupted .plist file was still the result of a hunch (and experience).

I’ve had DEVONthink crash a few times in the past, and have emailed the crash logs to Christian. Even in those cases, my database wasn’t damaged, although a couple of times I lost the most recently added item or two (I had preferences set to flush memory to disk every 3 minutes). Because I log DT entries in NoteTaker, it was easy to reenter the missing item or two and keep going. Note: DEVONthink Pro will flush new items to disk immediately, so it will be even harder to lose any data.

Most crashes have been pretty gentle (unlike the old days of Classic Mac OS). I saw more of them under X 10.2.x than I’ve seen under 10.3.3 – and most of them haven’t been DEVONthink’s fault (even when Christian was, by his own admission, torturing us testers with a rapid succession of betas).

Simple syntax checker for property list files:

% plutil -lint ~/Library/Preferences/com.devon-technologies.think.plist
/Users/sjk/Library/Preferences/com.devon-technologies.think.plist: OK

There’s a new GUI wrapper for plutil somewhere but I forgot its name and couldn’t find it searching for plist or property list on MacUpdate and VersionTracker.

Hard resets (e.g. after a kernel panic or system freeze or power outage) sometimes corrupt the last modified files (journaling does not prevent this), especially preferences and databases. Usually you don’t notice this in document based applications because those applications rarely access the disk. But even simple applications like iOrganize suffer from this problem.

By the way - the last fix of an severe I/O bug is around 9 months ago and all similar problems reported in the meantime have been related to the damaged file systems, kernel panics, system freezes, power outages, instable systems etc. and that’s of course something we can’t fix.

Only solution - do maintenance, keep your system clean and backup regularly. Sorry.

Is this because of OSX file system bugs?

I know databases like Oracle usually run directly on the
raw disk device so they control when and how things
are written. This gives them the ability to handle panics
and power failure while only losing uncommitted transactions.

On a product we are building at work, we are running
Postgresql on Linux ReiserFS and have had quite
a bit of trouble with the file system behavior during
power fail.  We are managing to hack around the problems
by detecting when blocks are incompletely written and
forcing file system metadata to be synced to disk
before file data.  Frankly, I don’t think we have found
all the holes, but statistically in our testing things are
getting better.

It’s not really a filesystem bug, it’s just a consequence of disk caching and that the cache sometimes couldn’t be completely written back to disk before a forced restart.

I’d really prefer if OS X would only cache reading and not writing but I guess that would slow down a lot of applications (e.g. the Finder) but not DT.

I appreciate the explanations, and think that the situation is clearer to me now.  While I appreciate that you may regard the problem I have experienced as an anomaly, let me try to summarize the objection I have to the approximate current behavior:

–Think is most useful if it may be left open to think into.
–A file which has been recently modified stands a chance of being lost if there is a crash affecting Think.
–Consequent to this, the files that one uses the most - in which the most frequently updated information is kept - are those which are most likely to be lost.

As a side note, I would mention that for many of us crashes are a routine part of life when working with certain kinds of applications (particularly real-time audio/video) or hardware.  A precision of my advice would be that if a user has routinely to work with such applications or hardware, he or she might find it prohibative to use Devon Think in the most logical way.

Best,
Yon

The next release will reduce the risk of losing data after a system freeze, kernel panic, power outage etc. as this release will force OS X to flush its disk cache back to disk everytime DT updated its database files (actually only the cache of the DEVONthink database folder will be flushed and therefore this won’t reduce the overall OS X performance)

In addition, DT Pro (public beta probably coming this month) will reduce the risk of losing data after a crash as this edition writes back all modifications back to disk immediately.