What might cause files to suddenly go missing?

I have just noticed that a number of my files within DT are now reported as missing. As far as I can see these were documents that I had scanned, and then added via DTTG. I recently upgraded to the latest version of DTTG. Would the old files somehow have been left behind in the previous version?

Would the old files somehow have been left behind in the previous version?

No. When DEVONthink To Go 3 launches, it copies the application data from version 2 to a new location.

Missing files in DEVONthink are most often caused by indexing files, then moving, deleting, or renaming indexed parent folders in the Finder.
Are you indexing files?

Not indexing, as far as I know. Just to be clear, indexing is when we reference files that are stored outside of the DT database, correct?

The file path of one missing document is
/Users/andyburnett/Databases/Family.dtBase2/Files.noindex/pdf/0/sample-report.pdf

Does that answer the question re indexing?

Cheers
Andy

Just to be clear, indexing is when we reference files that are stored outside of the DT database, correct?

Correct.

/Users/andyburnett/Databases/Family.dtBase2/Files.noindex/pdf/0/sample-report.pdf

Yes. That shows an imported file.

  • Is this an old database?
  • How recently have you done a File > Verify & Repair on it?
  • Do you have local backups of the database?

Thanks for the advice.
I ran Verify & Repair. The situation is much worse than I thought. It reported 93 missing files.

I do have backups. The problem is finding the point at which the files vanished. Are the files stored in folders, or are the rolled into a single database file?

On the face of it, it does seem to be safer to index files. I will do that in future.

Are the files stored in folders, or are the rolled into a single database file?

Files are stored in the internals of the database package, in the Files.noindex folder, segregated by extension.
We genrally advocate people do not mess about in there. In an emergency, it can be done provided primary backups are intact.

On the face of it, it does seem to be safer to index files. I will do that in future.

I would say that’s an inaccurate assessment :slight_smile:
Indexing isn’t the default behavior for a reason.
Indexing is discussed in the Help > Documentation > In & Out > Importing & Indexing, especially the Indexing and the file system section. Pay special attention to the Indexing and the filesystem section so you understand how actions in DEVONthink can affect files in the file system.

Thanks for the link. I see the limitations, and they are important to understand. However, if I understand the situation correctly, there is no simple way to recover the lost files once. The Time Machine backups will have stored the whole database, and so rather than being able to cherry pick the missing files, I would need to roll back the entire database. If that is true, then indexing does seem to be the simpler approach to dealing with recovering lost files.

The Time Machine backups will have stored the whole database, and so rather than being able to cherry pick the missing files, I would need to roll back the entire database

Yes, you would need to restore the whole database. However, you would restore it to a different location so you could open both and cherry-pick between them. When you’re done, you can delete the restored database.

OK, that makes sense. Thanks.

How would I craft a query to find the records that have missing documents?

Oh, the plot thickens!

It turns out that my desktop DT is reporting the files as missing, but when I look on my DTTG they are there. Any thoughts on what’s happening and how I fix this?

There isn’t a property for a file being missing and a missing file will still have a Path property.
Missing files are reported in Window > Log.

Is DEVONthink and DEVONthink To Go synced?

Yes, DTTG and DT are definitely syncing.

  • Is there anything reported in DEVONthink’s Window > Log?
  • Is there a triangle on the sync icon in DEVONthink To Go’s bottom toolbar? If so, touch it and tell me what it reports.
  • Are you using a shallow sync in DEVONthink To Go, i.e., Download Files: On demand?

The log file reports all the missing files - so that’s useful.

It seems as though a few files are not available on either system, which is disappointing.

No errors reported on the sync and yes I am syncing the full set of files.

  • Where is the database located - the file path on the machine?
  • On the off chance there’s a hardware issue, have you checked the health of the drive recently?

The path is

/Users/andrew/Databases/Family 3.dtBase2/Files.noindex/pdf/…
Disk utility doesn’t report any errors.

If the database is intact in DEVONthink To Go…
And it’s synced…
I would quit DEVONthink then delete the database on the Mac and reimport it from the sync location.
You can compress the .dtBase2 file in the Finder first, if you’d like.

OK, that sounds a bit worrying, but I will make a backup and give it a go.

Just for additional information, I have run a “Check file integrity” on the database and produced a report showing at least 50 entries had “Invalid File Integrity, checksum is different”.

Hmm, I am struggling to sync from DTTG to my Mac. How do I tell DT to sync a from DTTG when the database doesn’t exist on the Mac?

You sync with the sync location where DEVONthink ToGo syncs. you are not syncing with (directly) DEVONthink ToGo. The sync location is an intermediary storage location used only by DEVONthink products (not for mortals like us).

In DEVONthink, Menu: Preferences → Sync click pointer on the sync location. in the Database list (on right panel) at bottom, see “remote”. Your data base should be there shown being on the remote location. If not, click pointer on the “refresh”. When available, click on “import” under the Gear button.

1 Like