Problems Importing, Indexing, Losing data, Crashing, Copying

DEVONthink Tips
1

I tried using DTPO over the weekend again and it’s still not working reliably. I’ve had a number of problems including:

  1. Indexing taking forever, DTPO stops responding, reboot, rerun DTPO, 0 files indexed.
  2. Same problem with import: 0 files imported… folders and subdirectories are imported, but they are empty.
  3. Crashing (stops responding) when try to “run” the app.
  4. Have to reboot Mac over and over again to continue.
  5. Copying data from one DB to another doesn’t work.
  6. Seems the database is too “fragile”, meaning that if you do something wrong… or if you forget to do something, you risk corrupting the database. The only remedy seems to be to hope and pray that you have a backup (outsider DTPO, and “before” DTPO).
  7. Rebuilding the database leads to 1,000s of “Missing File” and “Failed” error messages.
  8. Importing gets stuck on file 759 of 38,946. Goto #4.
  9. Verification step finds 2 inconsistencies, 0 incorrect checksums, and 3,941 orphaned files. Then when I go into see the database, the orphan folder (AKA group) is empty.
  10. In the sidebar, next to the database names was displayed an icon that looked like an “X”. Zooming in on a print screen, it looks more like a pencil with a line through it / crossing it out. What does that icon mean, and how do you fix it?

This Mac has 4GB RAM. I’m using dtBase2 databases that range in size from 1MB… 700MB… 8GB… and up to 14GB. I’ve made lots of copies of the databases, and then I work on that copy. When that copy gets bad enough, I file it away and try again with another database. I think I need to throw these databases away and start over. Problem is, I’ve been doing that for a month now.

  1. The latest problem is that after trying to rebuild the biggest database (14GB) now when you click on a file, it shows the file “name” in pretty grey font, instead of the content. The files look like they are “there”, but they are “empty”. What caused that, how do you fix it, and how do you prevent it?

  2. Some of my databases are on the internal drive, others are on external drives, and networked drives. Can DTPO handle databases on different disks?

  3. Some of these disks are / are not indexed by Spotlight. Does Spotlight indexing affect whether or not DTPO works?

I would like to go on and dump more files into DTPO, but I’ve got to figure out how to make these smaller databases work first.

Am I the only one running into these problems?

I want to be your biggest fan. I see the potential for how DTPO could help tremendously simplify and speed up some of my work. This feels like a power tool that is dangerous in the wrong hands, but promises to be extremely powerful once you learn how to use it correctly.

2

DEVONthink Pro Office databases are normally very stable. But in the event of crashes, a database could be damaged, especially if data is still being written.

One of your problems may be with your networked drives. Check to make certain they are actually mounted when the database attempts to access them. Some servers are flaky on that point.

I notice that you are describing your databases by file storage size. That’s not the most important measure. The total word count, shown in File > Database Properties for each database, is far more important - and may have little relevance to the file size, as relatively small text files can have many times the word count of a large PDF file.

Although I’ve got an iMac with quad core i7 CPU and 8 GB RAM, I prefer working on laptops. My current favorite is a MacBook Air with dual core i7 CPU, 256 GB SSD and 4 GB RAM. In that environment, I use as a rule of thumb keeping the total number of words in a database (or the total number of words in a set of open databases) under 40 million total words. That limitation on total words leaves some RAM headroom for database work plus having a few other apps such as Mail, Safari or DEVONagent Pro open also. I like to have at least 750 MB of free RAM available, to avoid creating Virtual Memory swap files while I’m working, as that starts to slow down the computer. (I also try to avoid unnecessarily “hammering” the solid state drive by constantly swapping data back and forth between RAM and the SSD.)

Experiment with designing modular databases that meet a particular need. I usually have set open all the time, that includes my main database for research and writing (36.2 million total words) and others such as a financial database, etc. As needed, they can be treated like information Lego blocks, opened and closed if a different set is needed.

3

Bill, thanks for helping.

I like your suggestion to use RTF rather than PDF in order to get a cleaner file without all the ads or sidebars… I see now how that would help the AI search work better in the long run. Another approach I’m trying is the print view/icon that shows up on many of the news sites that I’m capturing from.

I have more information on the case of the copying not working. After rebooting and rerunning DTPO, I noticed something interesting. I hope this helps someone figure out what is happening or why it’s not working.

I tried this with two brand new databases. Database FROMHERE has nested folders with 62 files. I tried copying and moving them to the TOTHERE database. Used 3 methods to copy: dragging and dropping. right click and move to… right click and copy to. None of them seems to work.

The only thing that copied was the root folder, and one of the many subfolders.
I tried this a few times… created TEST1 folder in and tried copying in there. Created TEST2 folder and tried copying there. Created TEST3 folder and tried moving everything there. The only result was that the root folder copied along with subfolder. They were all emopty. 2 folders were created on the TOTHERE database, but none of the other sub-folders (groups) and none of the 62 files.

Then I tried something i read on this board about pressing either the command or option key (can;'t remember which!) AFTER I started dragging the contents. That trick seemed to work: i.e. the idea that you have to press the option or command or control key after you start dragging the mouse from here to there. All of a sudden, all the subfolders appeared on the TOTHERE database. And the files appeared. Or at least they looked ilk they were there: When you click any of them, I get the “No File” error message in the content window. NO FILE.

(Is part of the magic or secret that you have to press the key (what key :slight_smile: AFTER you start dragging the files over?)

I closed the databases… restarted DTPO… rebooted… ran rebuild… ran backup & optimize… nothing worked.

Today, I restarted Mac. Nothing was there in either database. My original files in FROMHERE are now gone however. So it looks like I’ve lost everything: Files are not in the FROMHERE db, and they are not in the TOTHERE db. The nested folders and files “look” like they are there in both databases.

But when I click a file in the FROMHERE db I get the dreaded "File missing: …path…filename… " error in the main content window: FILE MISSING.

And when I click a file in the TOTHERE db, I get the “No File” error in the main content window: NO FILE.

Then I look in the trash. FROMTHERE has 6 items in the trash. TOTHERE has 86 items in the trash!! To my surprise, it now looks like the files were copied or moved FROMHERE to TOTHERE after all… But they are in the trash. The file hierarchy looks good. I think it’s all there. When I gave up on this test yesterday I moved all those EMPTY folders in TOTHERE to the trash. But they were empty - - the copy and moves failed. I waited hours before trashing them.

Now all of a sudden the files are there in the TOTHERE db where i copied/moved them to. Is this normal, to wait overnight and reboot a few times before files actually copy or move?

What is going on? Are you supposed to reindex, reboot, rebuild, hope and pray, or what after you copy or move files?

4

I moved the parent folder from the trash into the database using the right click “move to” and selected the target database name (instead of the inbox or other folders). The move from trash into the database worked, and it happened instantly. That’s exactly how I expected the move or copy to work from one db to another: instantly. All folders and all contents moved to the new place.

I “'showed” some of the files in the finder before and after the move. It looks like the files are still stored in the same place in the internal DTPO database even after they are “moved.” Moving them to the right folder makes it “look” like they are right where I want them in the hierarchy. That works for me! At least they are out of the garbage and back in the database.

It’s still a mystery as to why the original copy or move attempts failed. Do you have to move to the “database name”, or can you move files to the inbox or any folder in the database? Something is up with copy/move.

Right now the only way I can copy/move files is to toss them in the trash, and then get my hands dirty taking them out.