Database inconsistencies after upgrade to DT3.6.3

My answers are in your comment.

Ok, so far so good. Do you use DEVONthink to go YES, and if so,
have you updated to version 3 YES, and if so,
did you update prior to version 3.0.3 being published YES and if so,
did you sync with your Mac YES?
But so far I did not encounter any problems, I have just updated the iOS devices to the most recent version, and synced test files going both ways (DT to DTTG and DTTG to DT), it works fine.
Furthermore, the 4th problematic folder contained an empty text file, maybe empty files cause the problem?

One more remark, I checked the empty files, their creation date and time is similar to those of the other files in the same folder. So I assume that they were there all along.

There was a problem in the initial implementation of DTTG 3 which could - for reasons not known with certainty, but based on current assumptions possibly related to a bug previously present in a past 2.x version, cause files to lose their content (i.e., go 0-byte in size). These files have been dubbed “ghost files”; a search for that term will bring up numerous reports here in the forum. It is conceivable that that is what happened here; it is also conceivable that instead a mechanism for detecting such files has triggered, although those files were perhaps previously empty.

The good news is that - as far a I know (and I think I am up to date on DEVONtech’s feedback to users on this) - once this has happened, it is not expected to happen again. So if you have solved the problem once, it should not recur. Whilst an automatic restore was implemented in current versions of DTTG 3, it did not work for every user; if you have, in fact, lost data, you’d probably have to restore it from a backup (if it’s lots and/or you don’t have a backup, there may be an alternative route via using DTTG 2; search for “ghost files”). Just to be sure, you might want to implement a smart group in DT3 which searches for kind is any document and size is 0. Also, in DTTG3 there is a smart group called “Ghosts”; it would show up any (other) ghost files (again, if that actually was even the problem here).

1 Like

Hello I have excatly the same problem except I have 24 inconsistencies.
2 issues:
-Some files are Endnote library - this I can deal with. I know EndNote and DT don’t sympatize too well
-It look that some of my pdf got split and are not usable anymore.

I just realised I can salvage pdf files by putting them in the finder and renaming them. But what a loss of time. Is there another way…Worrysome :roll_eyes

NB:
I do not use DTTG nor any cloud service with DT

Capture d’écran, le 2021-03-06 à 12.07.09

Capture d’écran, le 2021-03-06 à 12.09.22

@cgrunenberg coincidence or a mechanism for detecting ghost files acting up maybe? The last poster at least does not fulfil the criteria for even having ghost files as far as I can tell.

A ghost file isn’t necessarily from DEVONthink To Go or sync. With a few file formats filtered out as acceptable, zero-byte files are detected and reported as problematic.

That’s what I was guessing; so these files may have been there for aeons and are only being picked up now because a new mechanism for alerting when ghost files turn up has been implemented in DT3.0.x, right? That would mean that a small influx of people suddenly reporting problems does not necessarily point to a common source of the problem, just to a common mechanism of the reporting?

Correct. The maintenance routines have been improved and this is one byproduct.

1 Like

Excellent, thanks. Whilst that still leaves the three people who have reported problems in the last 24 hours with their problems (and we’ll all tune in to help), it does mean its not a systemic worry :+1:

Please can you explain in more detail what you mean by that? The PDF files in question, are they simply dormant in DT, or are they actively used by other software too? I’ve not heard of DT splitting PDFs into parts (it does have a split function, but that does not leave PDFs with .part extensions) so I’m just wondering whether this has anything to do with DT at all. The fact that DT is now drawing your attention to the problem does not infer that the problem cropped up recently (see the posts by Jim and myself above this one).

Hi Thanks for your replies.

About the splitted files:

In the log, among the 24 inconsistencies, I had several pdf that were reported as Empty File.
Ex: Schmitt_et_coll_2000.pdf

This file was dormant in DT. I had been used it for a medical expertise a few years ago and I probably had the file standing on my finder then I dragued it into a folder in specific patient folder in DT

When the check of the DB was made yesterday, the log was stating that it was now an empty file (before the last picture just up)

I went to locate it and in fact, there were 2 files: one of 0k and another (finissing by .part) of 370 k. None of these file were readable.

I tried to rename in with .pdf instead of .part - but it didnt work in DT
So I draggued both on my finder and I was able to rename the file with .pdf - it became readble (normal).

Just went through time machine before installing the new version: Eureka !

You were totally right !

This splitted files already existed in DV since 2020-01-07 (yesterday I forgot to check their modification date and had emptied my trask)

Interesting. After the last upgrade, I had one database that was trashed – hundreds of files missing or unreadable; it couldn’t be repaired. Fortunately, I have automatic cloud backup and was able to recover a pristine copy, but it wasn’t a happy experience. At the time, I suspected the upgrade, but couldn’t be sure since my other databases were fine. I ran Verify & Repair, on everything and they all passed, which was a great relief!

Using Verify and Repair I was able to fix inconsistencies manually, except for one in a specific database.
For this one - the log does not report anything. Empty line

QUESTION : How could it be ? It used to work fine in this same database for repairing the other inconsistencies.

@ DC Berk - happy for you it has been solved. Sometimes things get frightfull…

Please start a support ticket. Thanks.

done
Thanks Jim

After seeing a number of topics on this in the past few days I’m actually really concerned about this change in the maintenance routines. First of all I don’t agree that this is an overall improvement. Zero-byte files are first-class citizens and aren’t by themselves a cause for concern. In some cases are actually needed for certain applications.

As I understand it the reason for the change was that zero-byte files turned out to be a by product of a recent bug in DT/DTTG. In this situation it makes sense why you would disable sync in order to prevent the issue from spreading, but to me this feels like quite a desperate and dangerous solution. While it might improve the situation for the people that was affected by the sync bug, I can’t see how it could be an overall improvement of the software in it’s current form. In fact I’ll argue it’s the opposite.

The main problem is that this breaks sync silently for a number of people that haven’t had sync issues before with zero-byte files. Even if you decide to open up the log you’ll only be presented with “validation failed” errors. Nothing about sync not working. The only way to tell that something is wrong is by either noticing that notes are missing from synced databases, or by manually inspecting the time of the last sync.

I’ll be the first to forgive bugs in software, but I think in this particular case the user should have been better equipped to handle the issue. Rather than having to struggle to figure out why their sync stopped working, we should have been guided towards the problem and solution with proper help from the UI.

I really hoping the developers are on top of this, as I fear there might be a lot of unknowing users out there with non-functioning sync at the moment.

Not all zero-bytes files are detected as problematic. However, there a certainly times when they are so yes, they can be of concern.

Development will have to assess the situation and make their own determination on this.

Please provide an instance where you’re actively using zero-byte files. Perhaps legitimate use cases have been overlooked. Thanks.

I use them mainly for marking folders as cache or noindex directories, or for keeping track of running processes. I can definitely work around these use cases when working with DEVONthink, especially now that I know what the limitations are, so that’s not my main concern at this point. My primary concern is that blocked (by verification errors) syncs are not clearly communicated in the logs or in the UI, and that many users might be affected by this without then or us knowing about it.

By default Windows > Log is automatically shown (if not disabled).