Database inconsistencies after upgrade to DT3.6.3

Please start a support ticket. Thanks.

done
Thanks Jim

After seeing a number of topics on this in the past few days I’m actually really concerned about this change in the maintenance routines. First of all I don’t agree that this is an overall improvement. Zero-byte files are first-class citizens and aren’t by themselves a cause for concern. In some cases are actually needed for certain applications.

As I understand it the reason for the change was that zero-byte files turned out to be a by product of a recent bug in DT/DTTG. In this situation it makes sense why you would disable sync in order to prevent the issue from spreading, but to me this feels like quite a desperate and dangerous solution. While it might improve the situation for the people that was affected by the sync bug, I can’t see how it could be an overall improvement of the software in it’s current form. In fact I’ll argue it’s the opposite.

The main problem is that this breaks sync silently for a number of people that haven’t had sync issues before with zero-byte files. Even if you decide to open up the log you’ll only be presented with “validation failed” errors. Nothing about sync not working. The only way to tell that something is wrong is by either noticing that notes are missing from synced databases, or by manually inspecting the time of the last sync.

I’ll be the first to forgive bugs in software, but I think in this particular case the user should have been better equipped to handle the issue. Rather than having to struggle to figure out why their sync stopped working, we should have been guided towards the problem and solution with proper help from the UI.

I really hoping the developers are on top of this, as I fear there might be a lot of unknowing users out there with non-functioning sync at the moment.

Not all zero-bytes files are detected as problematic. However, there a certainly times when they are so yes, they can be of concern.

Development will have to assess the situation and make their own determination on this.

Please provide an instance where you’re actively using zero-byte files. Perhaps legitimate use cases have been overlooked. Thanks.

I use them mainly for marking folders as cache or noindex directories, or for keeping track of running processes. I can definitely work around these use cases when working with DEVONthink, especially now that I know what the limitations are, so that’s not my main concern at this point. My primary concern is that blocked (by verification errors) syncs are not clearly communicated in the logs or in the UI, and that many users might be affected by this without then or us knowing about it.

By default Windows > Log is automatically shown (if not disabled).

But to allow the spread of 0-byte files in users affected with the DT/DTTG sync issue is also undesirable. How do you separate what users are affected and who aren’t?

I’m also not sure how stopping sync to prevent unintended 0-byte files from spreading is ‘dangerous’ to those who weren’t affected, but that’s difficult to say as I don’t intentionally use 0-byte files. Do you really think many people do?

I agree that something like a pop-up with a consent form (‘click here if you understand the issue and want to sync anyway’) would be a more elegant approach, as it leaves the choice for the user to disable sync to prevent possible further damage or keep it running on purpose.

Right. Having the log icon on the menu bar definitely helps too (didn’t have that before). But even when inspecting the log it doesn’t tell you that sync isn’t working, only that some files failed verification. Failed verification log messages would not have been a cause of immediate concern for me had I not read these forum topics.

I’m not saying I have a better solution to the initial issue of disappearing notes and I fully agree that disabling sync for everyone possibly affected was most likely the best short term solution. Dangerous was perhaps too strong a word to use. Certainly propagating database corruptions are more dangerous than a failed sync. I’m just a bit surprised that a decision was made to stop sync for everyone without warning with zero-byte files, even those that had experienced no issues before.

I’m not trying to push blame or anything. I understand perfectly that these decisions are not easy. I just wish that we would get better errors or warning messages when syncing stops, besides “Failed database verification, please repair the database”. Just a bit more clear communication here down the line would work wonders, that’s all I’m saying. :slight_smile:

2 Likes

I’ve just had a similar experience. After reading this discussion, I ran Verify and Repair before updating from 3.6.2 to 3.6.3. That showed no errors or warnings, but repeating it after the update I found a few zero-byte files. I don’t use DTTG and I don’t sync databases.

If 3.6.3 is more rigorous in its checking, that seems a good thing :slight_smile:

2 Likes

I use LaTeX for creating slides for teaching and this often results in the creation of many zero length files. They are not necessarily critical but they certainly are legal and legitimate. That they might (do now) prevent sync’ing is a huge problem for me. Please don’t rely on this heuristic to identify and flag (irreparable) inconsistencies in a database.

As far as I know (but I could be mistaken of course), there isn’t another way to identify unintentional 0-byte files that seem to be corrupted by a yet to be identified cause and possibly spread across devices.

The question thus becomes: what should the DT staff rely on as an alternative to prevent possible data loss and when doen’t the precautionary action of disabling sync when 0-byte files are found outweigh the disadvantage to some users? Personally speaking, I’d rather have the DT staff to disable sync while they research the issue and keep files from corrupting even if that’s a disadvantage to sync.

If I’m not mistaken a request has already been made to disable this precautionary safety feature, but I’ll have to search for the comment where it was acknowledge to be sure.

If it’s really a temporary problem while they track down a bug then I’d think a temporary change to the repair database functionality could help. It could launch a dialog showing the 0 length files and gives some viable options at least? Right now we had to discover what the problems were by looking into the log (which is fine, that’s what it is there for) and then to find and delete these files (and then empty the trash, I think). Those steps could have all been more transparent, I’d have thought?

1 Like

Not just you. But if I’m not mistaken that was already mention in this topic :smiley:.

I have experienced similar issues since upgrading to 3.6.3 inconsistencies that repair can’t fix and after a rebuild. I even tried restoring from a version of database prior to upgrade and still shows inconsistencies. Seems like issues may have been in my database all along but previous version didn’t care. My main database also had inconsistencies flagged after upgrade but repair was able to fix these.

I have to agree with this comments. It took me a few days to realize the sync at stop from my main work computer to my backup computer. Only when I had to use the backup did I noticed files were missing. I reverted back to 3.6.2 on both which restarted sync and making sure all computers and iOS devices where on the same page (worked, in less then 2 min all where in sync nothing more to do!)…then start chasing the culprits based on the ghost folder on my iPad.

There are no reason that I can see from my perspective to stop sync. Warning and maybe moving the DTPO files on a Ghost folder also (like the iPad) would be a much better idea…