DTTG 3 is corrupting files

[Blanc]

@suavito

If I may barge in

you most certainly may @bosie is looking for ideas, and I’m not selling anything. The important point is that every user should have a backup strategy suited to the importance of their documents and the time they can invest in restoring. Thanks for sharing Let’s continue the discussion here.

[Solar-Glare]

If people don’t mind, I would suggest discussing backup strategies in another thread and leave this one for the topic on hand: do the files actually become corrupt?

Or are they (based on my quick and prelimary investigation) somehow duplicated and does one of the duplicates turn out to be 0-bytes in size, which are erronously presented as the primary record in DT?

[Taluyaq]

Some users have been discussing the same problem for a couple of days in another thread (Incomplete sync in DTTG3?). What you are suggesting is what I am hoping for, too. Maybe the corrupted files could be made accessible again.

[apb123]

I actually have had no problem at all. The new CloudKit syncing for me is rock solid. Using that smart group search didnt pick up any 0 size files or corrupted files.

I had been thinking Devonthink had been doing an amazingly good job and am really pleased with the reliable and faster sync.

Just a thought…but maybe you haven’t set the sync up correctly somehow ? It seems to work perfectly for me.

[Solar-Glare]

@apb123

maybe you haven’t set the sync up correctly somehow ?

Who knows, but if you’ve only seen white swans, that doesn’t mean other people could have seen black swans. I.e. you cannot deduct whether there is a bug present that causes this behavior because you didn’t experience it.

@Blanc: do I understand correctly you experienced the existence of 0-bytes files? Are you comfortable with closing DT completely, creating a copy (that’s important) of that erronous database and inspecting the content of the package? It’s not terribly difficult to understand the file structure, but as said be careful to not make any changes might you want to use that database file in the future as you could corrupt it if you alter it by accident. Hence the importance of creating a separate copy first.

If you’ve opened the package in Finder, browse to the content folder (it’s full with subfolders describing the various file types by extension like pdf or jpg) and search for the filename of the 0-byte file. If my findings are correct, you will find the 0-byte file in duplicate: one with a file size of 0 bytes and the original with a normal file size. As said I don’t know what the impact of opening the file could be, so please take into account any risks involved.

[Blanc]

No, I’m sorry if I was unclear in what I wrote; I have not experienced this problem - to the best of my knowledge my databases are intact.

(your instructions are immaculate - thank you for playing safe; I’ve previously looked at the file structure of DT databases. It was one of the things I did before committing to the software - I needed to be sure I could extract my data if all else failed).

[kewms]

@Solar-Glare

That said, it is my opinion that potential irreverisable problems (for those without backups for example) should be actively pursued as a high priority undertaking. That might be the case, but I don’t understand how the DT staff knows only a limited amount of users are affected. I certainly hope they don’t rely solely on users actively reporting this, as it’s something that might go unrecognized if a 0-byte file is tucked away somewhere in a database.

As someone experiencing this bug, I can say that, at least in my case, it only affects DTTG3. Neither the main database on the Mac nor the data in DTTG2 (on the same iPad) have any issues at all. At least in my case, it appears to be a problem between DTTG3 and the sync store, not a problem with the sync store itself, and therefore not a problem that will propagate to other copies of the data.

Not to sound insensitive, but it seems to me that people who don’t have backups for large amounts of critical data have a separate problem that DevonTechnologies can’t fix.

Katherine

[chdevonthink]

Oops! Hopefully a fix is soon found.

[eboehnisch]

We’re already working on a hot fix and will submit it to Apple tonight.

[Magusomo]

Will this fix only prevent “new upgraders” to get into trouble? Or will it also “cure” the actual corrupted files?

[Solar-Glare]

@eboehnisch

We’re already working on a hot fix and will submit it to Apple tonight.

That sounds like you gave it a high priority!

Any details on what went wrong and am I correct the files aren’t corrupted at all, but somehow get duplicated with a 0-byte size?

[psidnell]

I’d (obviously) be interested in what activity causes this.

I only use DTTG v3 and created a DB from scratch that syncs between iPhone/iPad via CloudKit and Bonjour.

So far all of my (several hundred) PDFs are fine as fine as far as I can tell.

[eboehnisch]

We believe that the problem occurs in a method that is applied to PDFs when you open them. It was actually suggested by the makers of the PDF engine and corrects highlighting and underlines that PDFKit on the Mac adds in a way that goes against the Adobe standard. When saving the corrected PDFs, for a yet unknown reason, writing the file fails and creates an empty or corrupted file. At least, that’s the theory for now as we have no device here that shows that problem in the labs.

We have now temporarily deactivated this method until a) we have made some further tests and b) added fallback code that keeps a backup copy of the PDF and jumps in when saving the PDF fails and replaces the corrupted copy with the backup.

Expect a 3.0.2 as soon as it has passed Apple’s review and a 3.0.3 later this week that hopefully reinstates the corrections.

What we don’t know is why it didn’t hit a single beta testers in many months and why it hits anybody at all because the code wasn’t changed in a long time. So we believe it must be a side-effect of the migration from v2 to v3 — but all beta testers went through that too. And it only hits some customers, not everyone.

When you receive 3.0.2 please let us know if it fixes the problem for PDFs you hadn’t opened before in DEVONthink To Go 3. The fix does not resurrect already destroyed PDFs. They are gone and you’ll need to replace them from your backup.

Please accept my apologies for this very unforeseen problem.

[Blanc]

Thanks for that update!

[Solar-Glare]

@eboehnisch

And it only hits some customers, not everyone.

I’ve mentioned this before, but how do you know whether or not it hit some customers and not everyone? Users don’t necessarily report 0-byte files if they’re only using DTTG for example or didn’t create a Smart Group like @Blanc suggested to present 0-byte files.

And when I inspect the DT database package, it appears that the 150-200 files or so files with a 0-byte size were duplicated in the package containing both the 0-byte version and the original.

[eboehnisch]

We don’t. We can just tell what we see in the feedback we get. If it hit everyone and no one except for a few people told us, we’re out of luck. And it definitely hit not a single beta tester as we hope they’d report the incident.

And when I inspect the DT database package, it appears that the 150-200 files or so files with a 0-byte size were duplicated in the package containing both the 0-byte version and the original.

What puzzles me: Did you open all these files in DEVONthink To Go 3 or are just randomly selected PDFs that went corrupt out of the blue? Here, too, a Console.log would be helpful. As it never struck us once here currently we can just build theories. What names did these two copies (zero bytes, intact) have? The more info we get from those of you who are affected, the better.

[Solar-Glare]

@eboehnisch

in the feedback we get

Working in a completely different field I’ve got some experience with this and in my experience feedback is not necessarily a good indicator of the width/size/scope of the problem.

Some people don’t report anything because they don’t mind, some people overreport and some only start reporting when they experience a sudden impact from a problem that long existed. But if feedback is all you’ve got to work width, I guess that’s all you’ve got.

@eboehnisch

Did you open all these files in DEVONthink To Go 3 or are just randomly selected PDFs that went corrupt out of the blue?

I didn’t ooen them, but what I do remember is that these files happened to be copied over yesterday from the global inbox to another database. So the order of operations for me seemed to be:

• DT3 and DTTG2 on multiple devices (baseline)

• WebDAV sync for all devices

• Upgrade to DTTG3 on two devices a couple of days ago (sync seemed without problems)

• Trouble updating DTTG2 because of family sharing issues on another device

• Moved 150-200 or so PDF files from global inbox to another database yesterday

• Bought a second license to perform the DTTG3 upgrade on other device

• Re-entered the sync details on that device as sync wasn’t transferred

• Some obscure error occured mentioning possible data loss

• Quit DTTG3, removed, reinstalled, synced and ended up with 1300 files in the inbox where all other devices report about 300

• Removed DTTG3 from that device

• Stumbled upon this thread and created a smart group as @Blanc suggested: scope all databases, size = 0

• Noticed the 150-200 or so PDF files I moved yesterday were reported as 0-bytes

• Read the tweet and didn’t touch the files

• Closed all instances of DT and removed all installations of DTT3 to prevent any further damage

• Copied most recent database containg the 150-200 0-byte files and a recent backup from my inbox (where those files originated from) and made a pairwise comparison on file level by examining the package content

@eboehnisch

Here, too, a Console.log would be helpful.

Without DTTG3 that’s not possible I’m afraid.

[eboehnisch]

So there was an error message? That’s a new factor. What did it say? And what a pity that we have no console log.

[Blanc]

@Solar-Glare

Some obscure error occured mentioning possible data loss

We may actually have to dissect you to see if we can find more details on that. I hope you don’t mind

[Solar-Glare]

Well, that happened this morning after some trouble with getting that device to connect to the sync store.

On the first devices everything seem to go without any problem and sync settings were automatically transferred. But on the other device that required family sharing, I removed DTTG3 awaiting 3.0.1 but left DTTG2. When 3.0.1 was released I installed it again hoping for a resolution of the family sharing issue, but it didn’t work (and downloaded through the family sharing option in the iOS app store as you suggested).

Ultimately I opted to,just buy another license after removing and re-installing once more. But then, quite unexpectedly the sync settings for WebDAV were not transferred (but probably not completely). Re-entered the sync settings but didn’t change the sync store name. Saw the different databases and started downloading the global imbox first (left the others untouched).

That is when the error occured: something about different content and possible data loss. Immediately quit syncing and removed DTTG3.

Omce again downloaded DTTG3 and started entering the sync settings, but now the sync store was changed to it’s default name ‘WebDAV’ (which it wasn’t the first time). After multiple trials discovered that the sync store name could be changed, changed that to the proper name and commenced syncing. When it finished syncing the global inbox, 1300 files were reported where all other devices report about 300.

That’s when I stumbled upon the thread about 0-bytes and started to delete DTTG3 from all devices and disabled DT awaiting your investigation.