So they were still fine on the Mac too? This would mean that the sync to DEVONthink To Go 3 actually destroyed them — or transported a damage that was made before.
Another question: Do you use an M1 Mac and with encrypted databases?
Switched from legacy to Cloudkit when moved from DTTG v2 to v3. No error. No index. My files affected are in Inbox, non encrypted. 2 PDF and 1 md files ended up with “0”
Were these files intact before you synced with DTTG3?
Actually, there’s no Mac involved in my setup anymore since 2019. I’ve gone completely mobile. But you are right regarding my observations. They point to a destructive process during the handling of PDF (meta?) data by DTTG 3. Whether it’s actively damaging data or just transporting damages I cannot tell from my surface viewpoint. Maybe also important: my sync store was always encrypted.
Yes, luckily I didn’t delete DTTG v2 on that iPad and checked the PDF there and it’s ok. Have 2 iPads.
Do you use a Mac? What about the files there? And do you, maybe, remember where you imported them, on iOS or Mac?
Mac’s corrupted. I used smart rule as posted by someone here to check and noticed 3 files with “0”. Imported on Mac. Very seldom import via iPad.
Eric, more info: PDF file was from January 21, 2020. PDF file still good on DTTG v2 but corrupted on both Mac v3.6.2 and DTTG v3.
@eboehnisch Can you check whether the 0-byte files in your database are duplicates of an original that is still in working order and ‘hiding’ within that database somewhere?
Or are your 0-byte files actually the only ones and do you consider the original data that the 0-byte file used to represent to be irretrievable from that database?
Edit: found three zip files that are not duplicates and thus would have been considered ‘lost’ if I didn’t have any backups. The duplicates happened to be present in the database as far as I can tell based on their name. So there goes my ‘duplicate’ theory.
what is the safest solution now for dt and dttg users? do not switch to cloudkit sync and stay with legacy? uninstall ddtg or not opening it enough until a bugfix has been released? checking the databases for 0 byte files each day? what would you recommend?
I’m equally in the dark as you are I guess. I was under the impression the 0-byte files were duplicates, but sadly discovered a couple of files that are indeed 0-bytes.
Personally I’m planning on removing the sync settings, reverting my databases to the version just before the date I installed DTTG3, purge my sync store, re-install DTTG2 as a work around and try to figure out what files were added or altered in the time in between.
And yes: I’ve setup a smart group looking for files that are 0-bytes (excluding groups)
Maybe stick to DTTG 2 and iCloud (legacy) (if that’s your current setup) until the guys at DT have a better understanding of what is going on.
The switch to CloudKit does not seem to be the problem (I base that on the fact that a poster further up reports to be using WebDAV only). I think it’s fair to say we (the folk in the forum) don’t know the cause or how wide-spread the problem is. I have implemented integrity checking and a smart group looking for 0-size files. I’m going to continue using DTTG3, and I’m trying to watch what I am doing meticulously so if a problem does occur I might be able to assist in pinpointing the cause. Ppl noticed the problems so early, that I think it’s possible whatever is happening either happens to you (when installing/setting up DTTG3) or it doesn’t happen at all.
If you currently have no problems, I can’t even be sure that the best advice is to put DTTG2 back and wait (whereas if I hadn’t got DTTG3 on my devices, I would currently refrain from installing it); as we don’t know what’s causing the problem, it’s hard to say what certainly won’t cause trouble… make sure you have up-to-date backups, a way to put them back, and a way to ensure they are still available in a few weeks time if it turns out there is something nasty going on. Don’t change your sync method at the moment.
1 Like
We are investigating the issue and don’t have a cause for certain at this time. We are also not receiving many support tickets reporting the issue at this time so much of the information is being anecdotally drawn from this thread.
Thanks to everyone who is constructively participating in it.
1 Like
Might it be wise to pin an official thread with updates on this issue on top of the dttg forum where only devontech publishes the current information and recommendation. it’s easier for customers to find the relevant information instead of going through all the threads. such an official statement / updates would help a lot. @eboehnisch @BLUEFROG
1 Like
Just to chip in a bit: I encountered this corruption once during beta tests. It was only one webarchive file so I assumed it was not a systematic issue. I also encountered issues where tags are not synced at all. I guess I should have reported these more diligently.
We have now uploaded a new build of version 3.0.2 to the App Store for review. Expect it to get automatically released in the next 24 hours, Apple allowing.
It addresses the problem pragmatically as follows:
- Before each sync it checks the database for empty files (excluding text and Markdown files from the check as they could be legitimately zero bytes).
- If empty files (‘ghosts’) are found the sync is not conducted but a new smart group activated that lists the affected documents. This gives you the chance to deal with them before the issue spreads through sync. It also switches sync to manual so new attempts are only made intentionally.
- Fixing the highlighting and underlining of PDFs coming from the Mac, as suggested by the makers of the PDF engine, is disabled for the moment as it might play a role here at least for PDFs.
- OCR in the background could create empty PDFs. We are intercepting this and don’t create a PDF at all.
This should prevent the damage from spreading and new damage for these potential causes from happening. And it should give us new insight into the issue.
There might be multiple issues at play but we are 100% sure that the actual data store migration doesn’t cause it. It’s a simple copy operation conducted by iOS itself, no files are actually touched. Similarly, deinstalling just version 3 and reinstalling doesn’t help as the shared container used by both generations of DEVONthink To Go is only emptied when both versions are deleted (and sometimes not even then but that’s outside of our control).
We will, of course, continue to try and find the root cause but as it happened, maybe, to some in the beta test we never had any reports in the past eight months. So we have to deal with this now in the real world.
Please let us know if this at least prevents the issue, finds the affected files, and especially if DEVONthink To Go produces new empty files.
5 Likes
No, I did not change sync methods. Yes, I did get the “couldn’t move to database package,” error, but I think not just for PDFs.
do I understand it correctly, you have the corrupt pdfs despite icloud legacy sync?
I use Dropbox sync exclusively, and have never enabled either iCloud legacy or CloudKit.
Note that it is not clear that my issue is identical to the one being discussed in this thread, as I haven’t seen the zero-length files anywhere except in DTTG3. In my case, at least, the corrupt files do not appear in either the source database on the Mac or in DTTG2.
Edit: Just confirmed this. @Solar-Glare’s smart group gives me no hits in any of the Mac databases.
Further Edit: I did do a Verify & Repair on the Mac databases as part of my troubleshooting for this. But it didn’t report any errors, so I doubt there was a trove of 0 byte files that the Repair removed.