Just add my experiences. I have had no problems at all. No corrupted files. I have 12GB over 6 different Databases.
The syncing is way faster in version 3.
1 Like
Well, I don’t feel the need, but from reading the posts I understood CloudKit would sync faster/better. But as I am not experiencing any problems, I first would like to know if switching to Cloudkit increases the risk of running into the corrupt files-problems.
I have about 6 databases, only a few GB but very important files that I invested in for many years to create. Ofcourse I create monthly backups so I can risk running into problems, but I would rather not 
Bonjour is fastest. do you use it or are in a position to use when both/all device(s) on your local network? if speed is what you want I would go for that in addition to your current method.
I am using the desktop-version 99% of the time. The iOS-version is just for when I can’t access my desktop-application. The only database I am syncing is a knowledge-database containing thousands of court decisions that I want to be able to access at any time. However, syncing indeed can be done at home using Bonjour.
1 Like
I’m surprised that my case of this type (I’d more accurately call it “DTTG 3 is corrupting files and/or failing to sync”), which I first presented as a help request related to my beta test release on 19 December 2020. Jim Neumann very patiently saw me through numerous tangles in this and finally told me to submit my problem as a bug report.
Here is the thread.
Dear Jim,
You will see the thread of our support discussion from last month below. Using your instructions, although I was unable to get the left-swipe to work for some reason (I simply deleted the Locations), I was able to restore the databases on my Mac from the intact databases on my iPad Mini, as well as on my iPhone, which only has one database on it, and my iPad Pro, although I’m not sure how complete it was on the Pro.
However, some problems remain. I have left my iPad Mini as it was, with DEVONthink to Go 2.7.8, since I know it is the original most complete version of the databases as they were on December 19. I experimented with updating DEVONthink to Go to the latest versions on the iPad Pro, and this has been problematic. As DEVONthink to Go 3 was accessing the databases, I get the error message “Error: couldn’t move [filename (mostly pdfs)] into the database package, as DEVONthink to Go 3 downloads updates from iCloud. There are a great many of them, and this paralyzes the app. How do I correct this?
Also, my workflow always begins on the Mac, and at least 98% of my files are entered into DEVONthink Pro Office on the Mac. I have been unable to upload these files to the iCloud location. Therefore it’s been several week since the files on the other devices have been updated. How do I solve this?
Once these issues are corrected, everything will be as it should.
Many thanks for your ongoing help with this.
Best regards,
Michael
On Dec 19, 2020, at 08:54, DEVONtechnologies Support <support@devontechnologies.com> wrote:
Dear Michael,
- Delete DEVONthink To Go 3 on any iOS device.
- On an iOS device with the disturbed database in DEVONthink To Go 2.7.8, go into Settings > Sync: Locations, left-swipe the active iCloud sync location, and choose Clean . This will remove the sync data. Then left-swipe the database on the Home screen and delete the problematic database. You’ll be importing it anew.
- On the Mac, delete the problematic database as you’ll be importing it afresh.
- In DEVONthink To Go 2.7.8 with the, enable the iCloud sync location, then sync the database. This will put intact sync data for the database in iCloud.
- On the other iOS device, import the database from the iCloud sync location. It should only be listed as Remote.
- In DEVONthink on the Mac, import the database from the Remote section of the Databases list when you have the sync location selected.
After these things are done, you may install the DEVONthink To Go 3 beta to check for the five upgrade options and to ensure the database migration worked as expected.
Best regards,
Jim Neumann
Customer Relations Specialist
Dear Michael,
It is quite possible iCloud has stalled - a condition over which we have no control, nor can we even detect it. We can’t reproduce this issue, and some have said it “just started working!”
Here are a few things people have said “worked” , though we obviously can’t test it since we can’t reproduce it. I would try them individually, starting with the simplest…
- Reboot the problematic device(s).
- In DEVONthink’s Preferences > Sync , disable the iCloud location and quit the application. Relaunch DEVONthink and re-enable the sync location again.
- In iOS’ Settings > your Apple ID > iCloud, try disabling and re-enabling iCloud Drive and DEVONthink.
- In iOS’ Settings > your Apple ID, sign out of iCloud and sign back in.
- In macOS’ System Preferences > Apple ID > Overview, sign out and back in.
No worries!
If the databases are intact on the Mac, I’d do this, again assuming the Mac has the master database(s)…
- Delete DEVONthink To Go, reinstall it, but do not set up the sync location yet.
- On the Mac, go into System Preferences > iCloud and click the Manage button. Select DEVONthink To Go then click the Delete documents and data button.
This preference doesn’t give a realtime view of the deletion process, so close and return to the Manage section occasionally to check the process. Once iCloud has removed the data from Apple’s server, the DEVONthink To Go entry will disappear. - In DEVONthink’s Preferences > Sync, right-click the iCloud sync location and choose Clean Location. This will clear out any sync data. You can now add, modify, or remove an encryption key for the sync location, but it must be done before you sync a database. This key would have to be entered in the sync location on other devices syncing with this location.
If you want to add/modify/remove the encryption key, you can now right-click the sync location and choose Show Info. Make the modification to the encryption key fields, the click outside the Info popup. - Then hold the Option key, control-click the iCloud sync location and choose Verify Location Thoroughly. The results will be shown in Window > Log.
- After these things are done, check the checkboxes next to the databases, in the Databases pane on the right, to sync to the location again.
- After a successful sync on the Mac, set up the iCloud location in DEVONthink To Go, using the same encryption key, if you specified one.
- Then touch the sync location and import the database(s).
DTSync is done locally, then iCloud uploads to Apple’s servers, then to devices using your Apple ID. This means databases may not be immediately available to sync on the other devices. The initial sync requires patience, as we have no control over the speed and reliability of iCloud’s process.
Also bear in mind, DEVONthink To Go should be in the foreground and the mobile device awake for the initial sync. The Background App Refresh option is controlled entirely by iOS and only allows approximately a 30 second window, when it allows it to happen. This means we can’t control if, when, or how long it happens. Subsequent syncs are faster since there’s less data being transferred.
Dear Jim,
I followed your directions in updating DTTG on my iPad Pro. In this I decided to add four small databases I didn’t include before. The result is that these four have propagated fully to the iPad, but the initial group of ca. 19 large databases appear as empty databases. You sent an earlier email about difficulties in uploading to iCloud, but I’m unable to find it. I have no idea how it could have been erased. Or perhaps you know some other solution.
I apologize for the long thread, but I hope it helps. I had no problems syncing before mid-December. As of now I have an almost-complete set of my files on my Mac, with four very large databases and 13 smaller ones. The one database that is missing is on my iPad Mini, the portable workhorse for most of the files, still in DTTG 2.87. I am reluctant to touch this installation, because it is my most complete set of databases, and I can’t afford to do without it. However, I am unable to update from my Mac, because sync doesn’t work. In the past I used iCloud Legacy for my attempts to rebuild the sync locations. I have now cleaned that location and am attempting a new start with CloudKit. This is now in process, so I can’t report on its success or failure. For these attempts, I have been using my less critical first-generation iPad Pro 12.9, which once had a nearly complete set of databases, but in syncing, it was repopulated with O byte ghosts of the files. It has DTTG 3.0.3 installed, which seems to offer no improvement. My iPhone has 3.0.3 with a single database, which is intact as of several weeks ago, but it still refuses to update. But in these two devices, I’ll have to report back, when my new location is complete.
No it’s not specific to CloudKit. And it’s not an issue everyone is experiencing.
Indeed. For what it’s worth I ended up with a bunch of ‘ghost’ files without using internet to sync.
It might be the synchronization somehow is involved, but I can definitely confirm ghost files (zero bytes) can occur in a situation where syncing to/from a WebDAV server is performed on a local network.
Whether it actually is the sync I cannot tell, but the ghost files did spread across devices. I’ve fairly accurately managed to pinpoint the first appearance of zero byte files in my Time Machine backup on the day I upgraded from DTTG2 to DTTG3 (I think this was in the morning). Those files weren’t present in the middle of the night but were present in the early afternoon. That means they either occured immediately after the upgrade, or after a couple of hours following the upgrade and/or sync.
No, unfortunately it did not.
For what its worth, here is my experience with this bug:
- Updated from DTTG 2 to DTTG 3.03 (had waited a week)
- I use an iPad, iPhone and Mac
- iPhone update seemed to go flawlessly, but iPad update did throw me the error and create a Ghosts folder with just a couple files in it
- This was cause for alarm, but I did not know about this thread and plugged forward…
- I deleted my databases on the iPad and synced anew via Bonjour
- Not sure my exact sequence but due to other ongoing sync issues I have had with Dropbox I decided to set up a brand new sync store on Dropbox. Did this on the Mac and let it all update.
- Setup new sync store on my DTTG devices and let it all sync (existing data was still on my iphone and ipad, so it merged), Ghosts no longer appeared and all seemed to sync fine.
- I have had ongoing issues with sync, using both Bonjour and Dropbox for years, so decided to limit my sync methods to just Dropbox to isolate potential issues, and remote sync is important to me so chose Dropbox. Turned off Bonjour on Mac and all DTTG devices.
- Saw this thread this morning…
- Setup a “0 size” smart group on Mac, and found 5 files in it, one being one of the PDF’s that was a ghost on the iPad (1 PDF and 4 tiny plain text docs)
- No Ghosts showing up on the DTTG devices
- Though file is registering 0 size, it actually IS there and can be opened, and in Finder it is normal. I copied it from the Finder to Desktop, deleted the one in DT, then reimported it back, the ghost disappeared on Mac. This makes me think that part of the issue is DT metadata about the file, as this PDF had content but was showing up as 0 bytes.
- The plain text files I “fixed” by either recreating them, or some were just one line notes so the title was the same as the content, so copying the title and pasting it into the content of the file seemed to fix it, at least it created a file with size.
- everything else seems OK that I can tell, but this is concerning…
…have Time Machine running constantly… but would be nice to be able to make sure I don’t have other issues I can’t see.
1 Like
I haven’t finished the slow process of uploading all my databases to the new Cloudkit location, but I have tested the small and large databases already there, and they are downloading successfully and seemingly complete to my iPhone and iPad Pro. Finally some progress, and possibly a solution! Vielen Dank, meine Herrn!
An update. I originally thiought I only had about 200 files but over the course of the last few months more and more keep showing up. Enough that I no longer use DTTG at all for fear of corrupting stuff and am moving things out of DT on the Mac as fast as I can. In the more recent problems I have at least been able to go back to my backups and find the files and recover them but the bug is still there.
I’m using WebDAV to sync and am running the latest versions of DT and DTTG.
more keep showing up
Please clarify this: showing up how and where?
Database on Mac is imported files. I have a smart group searching for zero length files that show up. I was still using DTTG on both iPad and iPhone and syncing via my own WebDAV server. I had more empty files show up after accessing the database in DTTG although I don’t believe I made any changes to those files. Fortunately as soon as I saw empty files I was able to go and recover them from backups from my mac. The data loss propogated across all devices before I could recover from the databse on the mac that was the current copy so I had to roll back to a prevoius days backup to recover the files.
So I’ve stopped using DTTG at all now because IMO it’s unreliable and can cause data corruption.
The first, primary and major function of a data repository like DEVONThink is not to mess up or lose my data. I can’t depend on it not causing me problems anymore. I have many thousands of files in DT and may not access them all that often. If I wasn’t checking for zero length files all the time I might not have seen the problem until it was too late. I can’t afford to be a guinea pig for the software and I can’t provide much of the data you ask other folks for about the problems as the information is all proprietary and cannot be shared. So for me the easiest and fastest solution is to quit DEVONThink. I’ve been using it for over 10 years but if I can’t depend on it, however painful the conversion is, I need to do it.
tbh, that comment does make me wonder… DT has a simple export function which will easily and quickly export the contents of your database to a virtually identical folder structure in Finder.
1 Like
Correct, but I am replacing the database functions I got with DEVONThink in Obsidian. So I need to convert to Markdown. And yes I can do that in DT too but I still have to get the structure re-created in Obsidian. I am doing that by using notes with links instead of folders and adding the crosslinks as necessary. So it’s not just a simple move the files and I’m done. I’ve got about 28,000 documents to move out of DT and into Obsidian.
thx for explaining
I had this exact same thing happen to me yesterday. I finally switched to DTTG3 last week, while on vacation, and set up a CloudKit sync store (I was using CloudMe before).
When I accessed my primary work database yesterday, I found around 70 PDFs stored in the Inbox of that database to be zero byte files, that were no longer accessible. Fortunately, I had switched off synchronization prior to updating to DTTG3, so the files on my Mac were not affected and I was able to recover them.
Here’s what I can tell you about the files that were affected. All of these were PDFs that were downloaded from a legal database and saved to Devonthink via “Save to Files” and then stored via the Files app directly in the Inbox of that database (the website offers PDF export, but DTTG had trouble storing the files via the Share sheet). When I saved these files, I was still using DTTG2. Sync store was CloudMe, as mentioned already.
Frankly, this is a huge issue in my opinion, that has shaken my confidence and faith in Devonthink quite significantly — currently, I feel that I can’t trust your software. I also find it quite worrying that months after this issue was first described, it still hasn’t been resolved. For now, I have decided to move away from the Devonthink ecosystem, which is a shame, because I’ve been a long time user and definitely like the apps.
1 Like
A couple of things: what you have experienced is the same as a relevant number of other users experienced, and seems to be related to a problem which long predated DTTG3; it’s likely impossible to resolve. What DEVONtech have done is to mitigate the problem: even if you had had sync turned on, it would almost certainly have terminated with an error message rather than propagating the problem to your Mac. This safety measure was implemented quickly after the problem of so-called ghost files appeared.
Once you have resolved the problem once, I would not expect it to return. @OogieM’s report of continuous propagation is the first I have read, and I’m not convinced they are reporting the same problem or mechanism. I note they are using an individual WebDAV setup for example.
It’s not my job to restore your confidence in anything, and I’m not trying to make little of the issue; DEVONtech themselves were devastated when data loss occurred, although they did find a workaround which automatically restored files for many users. I was worried about the issue at the time; I no longer am. It seems reasonable to assume that the only reason you have experienced the problem now is your switch to DTTG3; it’s also reasonable to assume you won’t experience it again. Again: everybody would have been happier had it never occurred; no software can guarantee data safety, however - both hardware and software can fail. As such, a comprehensive backup-strategy is a must.
In addition to a backup strategy, I have implemented a checksum routine - the idea being that I could otherwise, in theory, miss that files have changed and that years later my backups might no longer cover the appropriate time period.
Precautions such as these are requisite, regardless of which company & software you entrust your data to. DEVONtech reacted quickly to the issue described in this thread. Mistakes, unfortunately, happen - but there is something to be said for trusting those who own up to them, rapidly mitigating what has happened.
3 Likes
As we have explained before we believe that the root cause of this issue lies many months in the past and lurks in the sync stores. Therefore we have put multiple guard methods into place in both DEVONthink To Go and DEVONthink on the Mac that should immediate stop the sync in case ghost files are detected. Unfortunately in your cases the ghosts seems to have slipped the safety nets we have put in place and we can just apologize for the problems you are facing.
We continue to work on making DEVONthink and its sync to be as failure-proof as possible but still encourage everyone to make regular backups as not only DEVONthink can fail but also all other parts of the system from the hardware to te operating system itself.
1 Like