DTTG 3 is corrupting files

DEVONthink To Go
249

Thank you for this suggestion. I did the same thing and have 120 empty, some from years ago. I have been asking Devonthink for years to provide a mechanism for verifying the integrity of the database. Now I have a day or so work going back though ancient backups looking for these files.

I love Devonthink but I do not trust it and have no way of verifying the files. Until they put in a mechanism (like git) to validate the database using a cksum against each file I am done.

I’m looking for an alternative solution, any ideas? Ill probably just go back to directories of files protected with a CKSUM. It is more reliable.

Oh … and I had to pay for this cr@p.

Not at all happy.

1 Like
250

Bear in mind it’s certainly not said that it’s some bug DT could have foreseen or even take into account. For all we know it could be the result of some ‘ghost’ from the past that came back to haunt us, or a change in iOS that DT has nothing to do with. I understsnd they’ve beta tested the program for a long time and apparently the staff wasn’t aware of this problem and I presume they wouldn’t have released it if they did.

To be honest, I think there is no document system you can ‘trust’ as all can somehow lead to loss of data, which is one of the reasons reason you need a backup.

And DT has picked up the situation quite fast, creating a way that shows you whether or not you have ‘ghost’ files and tries to automatically restore them.

If DTTG2 was installed and you installed DTTG3 in parallel (i.e. both apps are or have been on the device at the same time) the DTTG2 data store is likely intact. If you’ve removed both DTTG2 and DTTG3 the data on that device is deleted though. Did you update to version 3.0.3 of DTTG3 (meaning you leave a prior version on the device and update through the app store)? Because it’s likely the ghost files can be repaired if at least one of the versions has always been on the device.

3 Likes
251

I had a problem before I was aware of this thread. It related to metadata in pdf files. In my case this was tags. I have items that are receipts and each has a set number of tags and sub tags within those main groups. I can quickly see problems if the top level tag counts aren’t all the same and this happened on my iPhone running DTTG 3 the other day. What happened is that while the Mac showed no problems, the iPhone started showing reduced counts. What had happened is that one sub tag was no longer present, but there was a group with the name of the sub tag sitting in the top level of the database. In that group were all the receipts that had the missing tag. I created the tag again and drug those files to the newly created tag and all was well.

Until I turned on my iPad and suddenly a different set of sub tags disappeared. This time, I found the two groups in the trash. Again I was able to recover but I was shaken.

I deleted DTTG from my iPhone and iPad and made a clean install of DTTG3 on both devices and let them sync from the Mac. So unfortunately I don’t have any log files to share.

In my case syncing is over the local network and that didn’t change between DTTG2 and DTTG3.

This may be a different issue than the one being discussed but since the conversation has moved to metadata I thought my experience might help shed a little light.

252

I use EagleFiler for business records. It stores files directly in the Mac file system and includes some data integrity features. It has many similar features to DEVONThink.

I also use DEVONThink Pro very extensively for various of my projects and researches and appreciate its particular features very much. Though I have never suffered any file losses using DTP over many years, I prefer not to rely on database systems where I need to be sure of long term access. Hence using a different tool for my business records.

253

I can relate, though I’m not leaving DT for the time being. I still like it a lot for the ingest and processing stages, but I do have a secondary archiving solution (which is based on git-annex, by the way).

I do agree that checksumming should be implemented in some fashion. It can be an expensive operation (though not as bad with some modern hashing algorithms, like xxhash which would probably suffice) so perhaps not enabled everywhere by default.

Perhaps something tied to locking? A locked file shouldn’t change, a way of detecting if this happens inadvertently would be very useful. It could be up to the user to decide whether to spend the CPU cycles.

254

you might be interested in this thread:

1 Like
255

Hi, I am fortunately not experiencing any issues until now and I am thinking about changing from iCloud legacy to Cloudkit. I am not sure yet when users are likely experience the corrupt files-problem and I am wondering if switching syncing from iCloud to Cloudkit could increase the chance of running into problems. Can anyone tell me if the issue is related to Cloudkit?

256

if no issues, why change? what is the compelling treason?

257

Just add my experiences. I have had no problems at all. No corrupted files. I have 12GB over 6 different Databases.

The syncing is way faster in version 3.

1 Like
258

Well, I don’t feel the need, but from reading the posts I understood CloudKit would sync faster/better. But as I am not experiencing any problems, I first would like to know if switching to Cloudkit increases the risk of running into the corrupt files-problems.

259

I have about 6 databases, only a few GB but very important files that I invested in for many years to create. Ofcourse I create monthly backups so I can risk running into problems, but I would rather not :slight_smile:

260

Bonjour is fastest. do you use it or are in a position to use when both/all device(s) on your local network? if speed is what you want I would go for that in addition to your current method.

261

I am using the desktop-version 99% of the time. The iOS-version is just for when I can’t access my desktop-application. The only database I am syncing is a knowledge-database containing thousands of court decisions that I want to be able to access at any time. However, syncing indeed can be done at home using Bonjour.

1 Like
262

I’m surprised that my case of this type (I’d more accurately call it “DTTG 3 is corrupting files and/or failing to sync”), which I first presented as a help request related to my beta test release on 19 December 2020. Jim Neumann very patiently saw me through numerous tangles in this and finally told me to submit my problem as a bug report.

Here is the thread.

Dear Jim,

You will see the thread of our support discussion from last month below. Using your instructions, although I was unable to get the left-swipe to work for some reason (I simply deleted the Locations), I was able to restore the databases on my Mac from the intact databases on my iPad Mini, as well as on my iPhone, which only has one database on it, and my iPad Pro, although I’m not sure how complete it was on the Pro.

However, some problems remain. I have left my iPad Mini as it was, with DEVONthink to Go 2.7.8, since I know it is the original most complete version of the databases as they were on December 19. I experimented with updating DEVONthink to Go to the latest versions on the iPad Pro, and this has been problematic. As DEVONthink to Go 3 was accessing the databases, I get the error message “Error: couldn’t move [filename (mostly pdfs)] into the database package, as DEVONthink to Go 3 downloads updates from iCloud. There are a great many of them, and this paralyzes the app. How do I correct this?

Also, my workflow always begins on the Mac, and at least 98% of my files are entered into DEVONthink Pro Office on the Mac. I have been unable to upload these files to the iCloud location. Therefore it’s been several week since the files on the other devices have been updated. How do I solve this?

Once these issues are corrected, everything will be as it should.

Many thanks for your ongoing help with this.

Best regards,

Michael

On Dec 19, 2020, at 08:54, DEVONtechnologies Support <support@devontechnologies.com> wrote:

DEVONtechnologies company signet

Dear Michael,

  1. Delete DEVONthink To Go 3 on any iOS device.
  2. On an iOS device with the disturbed database in DEVONthink To Go 2.7.8, go into Settings > Sync: Locations, left-swipe the active iCloud sync location, and choose Clean . This will remove the sync data. Then left-swipe the database on the Home screen and delete the problematic database. You’ll be importing it anew.
  3. On the Mac, delete the problematic database as you’ll be importing it afresh.
  4. In DEVONthink To Go 2.7.8 with the, enable the iCloud sync location, then sync the database. This will put intact sync data for the database in iCloud.
  5. On the other iOS device, import the database from the iCloud sync location. It should only be listed as Remote.
  6. In DEVONthink on the Mac, import the database from the Remote section of the Databases list when you have the sync location selected.

After these things are done, you may install the DEVONthink To Go 3 beta to check for the five upgrade options and to ensure the database migration worked as expected.

Best regards,

Jim Neumann
Customer Relations Specialist

Dear Michael,

It is quite possible iCloud has stalled - a condition over which we have no control, nor can we even detect it. We can’t reproduce this issue, and some have said it “just started working!”

Here are a few things people have said “worked” , though we obviously can’t test it since we can’t reproduce it. I would try them individually, starting with the simplest…

  • Reboot the problematic device(s).
  • In DEVONthink’s Preferences > Sync , disable the iCloud location and quit the application. Relaunch DEVONthink and re-enable the sync location again.
  • In iOS’ Settings > your Apple ID > iCloud, try disabling and re-enabling iCloud Drive and DEVONthink.
  • In iOS’ Settings > your Apple ID, sign out of iCloud and sign back in.
  • In macOS’ System Preferences > Apple ID > Overview, sign out and back in.

No worries!
If the databases are intact on the Mac, I’d do this, again assuming the Mac has the master database(s)…

  1. Delete DEVONthink To Go, reinstall it, but do not set up the sync location yet.
  2. On the Mac, go into System Preferences > iCloud and click the Manage button. Select DEVONthink To Go then click the Delete documents and data button.
    This preference doesn’t give a realtime view of the deletion process, so close and return to the Manage section occasionally to check the process. Once iCloud has removed the data from Apple’s server, the DEVONthink To Go entry will disappear.
  3. In DEVONthink’s Preferences > Sync, right-click the iCloud sync location and choose Clean Location. This will clear out any sync data. You can now add, modify, or remove an encryption key for the sync location, but it must be done before you sync a database. This key would have to be entered in the sync location on other devices syncing with this location.
    If you want to add/modify/remove the encryption key, you can now right-click the sync location and choose Show Info. Make the modification to the encryption key fields, the click outside the Info popup.
  4. Then hold the Option key, control-click the iCloud sync location and choose Verify Location Thoroughly. The results will be shown in Window > Log.
  5. After these things are done, check the checkboxes next to the databases, in the Databases pane on the right, to sync to the location again.
  6. After a successful sync on the Mac, set up the iCloud location in DEVONthink To Go, using the same encryption key, if you specified one.
  7. Then touch the sync location and import the database(s).

DTSync is done locally, then iCloud uploads to Apple’s servers, then to devices using your Apple ID. This means databases may not be immediately available to sync on the other devices. The initial sync requires patience, as we have no control over the speed and reliability of iCloud’s process.

Also bear in mind, DEVONthink To Go should be in the foreground and the mobile device awake for the initial sync. The Background App Refresh option is controlled entirely by iOS and only allows approximately a 30 second window, when it allows it to happen. This means we can’t control if, when, or how long it happens. Subsequent syncs are faster since there’s less data being transferred.

Dear Jim,

I followed your directions in updating DTTG on my iPad Pro. In this I decided to add four small databases I didn’t include before. The result is that these four have propagated fully to the iPad, but the initial group of ca. 19 large databases appear as empty databases. You sent an earlier email about difficulties in uploading to iCloud, but I’m unable to find it. I have no idea how it could have been erased. Or perhaps you know some other solution.

I apologize for the long thread, but I hope it helps. I had no problems syncing before mid-December. As of now I have an almost-complete set of my files on my Mac, with four very large databases and 13 smaller ones. The one database that is missing is on my iPad Mini, the portable workhorse for most of the files, still in DTTG 2.87. I am reluctant to touch this installation, because it is my most complete set of databases, and I can’t afford to do without it. However, I am unable to update from my Mac, because sync doesn’t work. In the past I used iCloud Legacy for my attempts to rebuild the sync locations. I have now cleaned that location and am attempting a new start with CloudKit. This is now in process, so I can’t report on its success or failure. For these attempts, I have been using my less critical first-generation iPad Pro 12.9, which once had a nearly complete set of databases, but in syncing, it was repopulated with O byte ghosts of the files. It has DTTG 3.0.3 installed, which seems to offer no improvement. My iPhone has 3.0.3 with a single database, which is intact as of several weeks ago, but it still refuses to update. But in these two devices, I’ll have to report back, when my new location is complete.

263

No it’s not specific to CloudKit. And it’s not an issue everyone is experiencing.

264

Indeed. For what it’s worth I ended up with a bunch of ‘ghost’ files without using internet to sync.

It might be the synchronization somehow is involved, but I can definitely confirm ghost files (zero bytes) can occur in a situation where syncing to/from a WebDAV server is performed on a local network.

Whether it actually is the sync I cannot tell, but the ghost files did spread across devices. I’ve fairly accurately managed to pinpoint the first appearance of zero byte files in my Time Machine backup on the day I upgraded from DTTG2 to DTTG3 (I think this was in the morning). Those files weren’t present in the middle of the night but were present in the early afternoon. That means they either occured immediately after the upgrade, or after a couple of hours following the upgrade and/or sync.

265

No, unfortunately it did not.

266

For what its worth, here is my experience with this bug:

  • Updated from DTTG 2 to DTTG 3.03 (had waited a week)
  • I use an iPad, iPhone and Mac
  • iPhone update seemed to go flawlessly, but iPad update did throw me the error and create a Ghosts folder with just a couple files in it
  • This was cause for alarm, but I did not know about this thread and plugged forward…
  • I deleted my databases on the iPad and synced anew via Bonjour
  • Not sure my exact sequence but due to other ongoing sync issues I have had with Dropbox I decided to set up a brand new sync store on Dropbox. Did this on the Mac and let it all update.
  • Setup new sync store on my DTTG devices and let it all sync (existing data was still on my iphone and ipad, so it merged), Ghosts no longer appeared and all seemed to sync fine.
  • I have had ongoing issues with sync, using both Bonjour and Dropbox for years, so decided to limit my sync methods to just Dropbox to isolate potential issues, and remote sync is important to me so chose Dropbox. Turned off Bonjour on Mac and all DTTG devices.
  • Saw this thread this morning…
  • Setup a “0 size” smart group on Mac, and found 5 files in it, one being one of the PDF’s that was a ghost on the iPad (1 PDF and 4 tiny plain text docs)
  • No Ghosts showing up on the DTTG devices
  • Though file is registering 0 size, it actually IS there and can be opened, and in Finder it is normal. I copied it from the Finder to Desktop, deleted the one in DT, then reimported it back, the ghost disappeared on Mac. This makes me think that part of the issue is DT metadata about the file, as this PDF had content but was showing up as 0 bytes.
  • The plain text files I “fixed” by either recreating them, or some were just one line notes so the title was the same as the content, so copying the title and pasting it into the content of the file seemed to fix it, at least it created a file with size.
  • everything else seems OK that I can tell, but this is concerning…
    …have Time Machine running constantly… but would be nice to be able to make sure I don’t have other issues I can’t see.
1 Like
269

I haven’t finished the slow process of uploading all my databases to the new Cloudkit location, but I have tested the small and large databases already there, and they are downloading successfully and seemingly complete to my iPhone and iPad Pro. Finally some progress, and possibly a solution! Vielen Dank, meine Herrn!

270

An update. I originally thiought I only had about 200 files but over the course of the last few months more and more keep showing up. Enough that I no longer use DTTG at all for fear of corrupting stuff and am moving things out of DT on the Mac as fast as I can. In the more recent problems I have at least been able to go back to my backups and find the files and recover them but the bug is still there.

I’m using WebDAV to sync and am running the latest versions of DT and DTTG.