Deleting a mega-database

Strictly speaking, this isn’t really a DT issue, but I’m hoping some of the knowledgeable people here may have some idea what to do.

Since switching to DT Pro, I haven’t had any of the slowdown issues that I used to have with early versions of the Personal edition. Added to the fact that the Finder is very slow to calculate the size of large files, I completely failed to realize until recently how big my main DT Pro database was: 250 GB, which takes up most of my internal hard disk. I copied the database over to a bigger, and empty, external FireWire drive, bought for this purpose, and moved the original on my internal drive to the trash, with a view to freeing up space.

Whenever I try to empty the trash, however, the screen goes dark and I receive a message telling me to restart the computer using the power button. After restarting, the error report says “panic…too big” or something to that effect. Is there anything I can do to get rid of the file and actually free up space on my internal drive?

(Note that I figure it probably isn’t healthy to let databases get this big, so in the medium term I’m planning to cull some documents from the database on the external drive, and export some parts of the database to new databases. But in the short term I really want to get rid of the huge file.)

I’m fascinated. What kinds of files are in your database?

Suggestion: Try moving the big database out of the trash, e.g. to your Desktop. If it hasn’t been damaged, see if you can open it. (Make certain its the copy that you want to delete, not the copy on your external drive.)

Then delete selected groups incrementally until the database is empty, or it’s small enough to delete without a Finder or OS error. Note that some file types such as PDF, images and QuickTime media (in other words, the file types that were stored in the internal Files folder) will be diverted to the Trash, and you will need to frequently empty the Trash.

I used to have mainly journal article PDFs, but recently I’ve imported some whole websites (one or two of which took hours to import, so I guess they must have been pretty big) plus a set of Paparazzi screenshots of all my del.icio.us bookmarks.

Once I’d taken the time to describe my situation on the forum, the idea of moving the database out of the trash, opening it, and deleting files, occurred to me. Unfortunately, the bit about files being diverted to the trash didn’t, so now I have a trashcan mega-full of files (which I can’t empty in one go). I’ve started creating a series of folders on the desktop called trash1, trash2, etc., and I’m going to try moving them one by one into the real trash, emptying it each time. What puzzles me is that, although I’ve deleted pretty much everything, the database shows as being 215GB large in the Finder. Does that mean it still “contains” the material that hasn’t yet been erased from the disk?

Another thing that confused me when deleting files from the database was the message asking me whether I wanted to delete from the database only or also from disk. This is relevant to my typical usage scenario: I have a folder in my Documents folder called Import to DT, with the relevant script attached. It seems unlikely, but would DT actually go into the Finder and delete these original files?

Anyway, I’ve decided to take a few days off for New Year, and will tackle this situation again when I get back. Thanks for your help,

Rick

Rick, if you confirm that files should also be deleted from disk that will happen. Otherwise, they will only be deleted from the database.

Revisiting this issue, I thought I should let you know that I think I’ve isolated the problem.

I managed eventually to delete all the ordinary files that had filled up the trash as I purged my database. That left a very large DT database. After taking it out of the trash I opened it, and delteed all the files, remembering this time to empty the trash frequently. I eventually got it down to empty, but was alarmed to see that the Finder thought it was still 211 GB. Attempting to trash this and empty the trash brought another kernel panic.

Right clicking, I inspected the package contents of the database, and found that, amongst other things, there was a single .mov file of 211 GB (screencast iWeb ED.mov, to be exact). I deleted all the other little files, but I’m at a loss what to do to get rid of the .mov file. I’m sure it can’t have been that big when I downloaded it!

Does anyone have any ideas for getting rid of this file to free up some disk space?

For delete files from an uncooperativeTrash, you can let Batchmod a try:
macchampion.com/arbysoft/

If this not works…
a. Put the problematic database onto the Desktop; I suppose its name is perso.dtbase
b. elsewhere but not on the Desktop, for instance in your Documents, create an empty TextEdit file with the exact filename (including the extension) than your database; TextEdit can warn you that the .dtbase is not suitable for text; force it to accept the extension
c. Drag this file onto the Desktop; of course Finder will ask you “Do you want to replace…blabla…”; and of course accept;
d. now, on your Desktop a simple nearly 0 bytes dummy database you can delete

Well, if this not work, we can think about the Terminal and a command to do the dirty job, but Terminal, in Superuser mode, can be dangerous
For this, see the XLab: thexlab.com/faqs/trash.html

Thanks a bunch for the advice. I figured, wrongly it turned out, that the 2nd method would be the safest. When I gave permission to replace the real file with the dummy created in TextEdit, I got another kernel panic. Then I tried BatChmod. As soon as I changed permissions on the file, its size, according to the Finder, changed from more than 200 GB to 4KB, and it was readily deleted. But checking the hard disk information, the amount of available space hadn’t changed that much.

So I guess the file never was all that big, just corrupted in some way that fooled the Finder into thinking it was big. I’m disappointed in the sense that I’m probably going to have to do a real culling of my files. But of course that is far outweighed by the relief that my trash can now be emptied safely, and the loss of that vague sense of unease at the existence of a file that I can’t understand!

Kernel panics after deleting a file sound like a damaged filesystem. Checking (and repairing) the filesystem (and the permissions of the startup volume) is highly recommended.

Repair procedure: thexlab.com/faqs/repairprocess.html
Running maintenance scripts : thexlab.com/faqs/maintscripts.html
Tool for maintening: Onyx : titanium.free.fr/pgs2/english/onyx.html

Track big files:
WhatSize: id-design.com/software/whatsize/index.php
DiskInventory: derlien.com/
Omnidisksweeper: omnigroup.com/applications/omnidisksweeper/
(demo mode only allows to localise big files)

This probably sounds rather ridiculous, but yesterday I found the offending file on the desktop again. I can’t imagine why; maybe I got confused with the dummy file I had created in an attempt to replace the offending file.

Neither BatChmod nor Onyx were able to help, restarting from a system disk and doing a hardware check and then running disk utility accomplished nothing, and trying to delete the trash through the Terminal likewise simply brought on another kernel panic.

In the end, a more focused Google search (empty trash kernel panic Tiger) found that, in Tiger, this is a somewhat known problem, and that it’s connected to disk journaling. By turning that off temporarily the problem was easily solved. Now the file appears to be well and truly gone and a Get Info on my disk shows that I’m only using a total of 110 GB (whereas that one file, hidden away inside a DT database, was taking more than 200 GB all by itself.

This is really good, because I had given up cloning my startup drive regularly, as it was too big to fit on the external drive I bought for the purpose.

Anyway, thanks to everyone for all your help,

Rick

http://www.thexlab.com/faqs/jnlrelatedpanictiger.html