As far as I understand the problem seems to be a metadata issue since DTTG 3 does not actually download whole files during migration. Therefore it would be reasonable to assume the corrupted files might still be accessible from the sync store but need to be re-read completely by v3. For those files already in the ghost group such a strategy could be implemented as a voluntary fix in one of the next updates because doing more harm on them is unlikely.
1 Like
@eboehnisch Eric, I installed 3.02 this morning and I have two PDF documents in the Ghost smart group, and the file info for each shows the size is 0 bytes. However, the documents appear to be fine-one is 18 pages and the other is 24 pages, and all the content is normal. I donāt know if this info is helpful at all, but I wanted to pass it on just in case.
I can add to that the following:
Iāve fairly accurately pinpointed the time when the 0-bytes appeared, based on an inspection of TM backups of my global inbox.
One of my 0-byte files was a noticeable 50MB zip file that was (and should be) present in the inbox. Itās also the file that is still available in DTTG2 I mentioned above.
Feb 10
release DTTG3 (is this correct?)
Feb 10
updated two of my iOS devices with DTTG3
Feb 11-12
trouble using family sharing
Feb 13
bought second DTTG3 license
synced that iOS device
sync issues with some error about data loss
removed DTTG3 from that device
Iāve now copied the Inbox packages from Time Machine dating from Feb 10 at 1AM (I was asleep), Feb 11 and Feb 15. What I found was the following:
- All inboxes have an identical size in Finder (about 400MB)
- The Feb 10 Inbox holds the 50MB zip according to Finder
- The Feb 11 and 15 Inbox show the zip as 0-bytes, but the total Inbox package size has not changed
Update: the package size is clearly wrong. Finder correctly calculates the expected 50MB difference in the Files.noindex folder when I compare the packages that were stored in Time Machine before and after February 10th. So:
- the packages all show about 400MB
- the Files.noindex folder has decreased as expected by 50MB in the package of February 11th
Update2: according to Finder in the package of February 11th (one day after the upgrade) three zero-byte files were modified on February 10th around 13:55. A file I happened to annotate by accident was changed on February 10th at 13:54. The metadate file was changed on February 10th at 14:01.
2 Likes
Thank you for letting us know. Thatās just wrong metadata. Version 3.0.3 will repair that on the fly.
1 Like
Any chance that you could send us zipped copies of all these inboxes? This might help trouble-shooting the issue.
During the migration of the old data store no data is downloaded at all. What happens is that V3 makes a copy of the folder that contains the database and all accompanying files. With APFS that just takes a fraction of a second and consumes no additional space. It then uses the copy, migrates the database scheme, and thatās it. Individual files are, at least not intentionally, touched. All error messages like āCouldnāt move ā¦ into the database packageā and similar are not part of the migration from V2 to V3 but happen later, as part of a sync.
Thatās why our theories go into the direction that the problem is actually caused by problems in the sync store itself that is then downloaded to DTTG3.
If anyone of you has a sync store on WebDAV or Dropbox that, presumably, holds the problem: Could you share that sync store (.dtSyncStore folder) with us so that we can have a look?
That sounds correct as DEVONthink To Go 3 is technically copying the version 2 data store. But as itās an APFS copy (Appleās new file system) it doesnāt physically use all that space.
I would, but I do not find it. Using WEBdav on a Synology with DSM 6.x. In my DEVONthink folder, I have
.DS_Store DT.dtCloud @eaDir #recycle
and in DT.dtCloud I see
09b8bafd2d664164cac0aeab533876724187c0f65e56255174ce5f25903b375c
4071363aacbeb8ea1ad26584a4ec9463c3fa46d3bbc31acf0ac0955569ba9883
7cba81b0c71fc3f8b7135c7d6a933cd51baecb26d8c3cf66c23197c51db572e5
7ce2ed434f942e4d75d0a25aa5a3cb663163042e75cf948dc3ada0ec4f1c7a29
9e99eeb4607550a68d0c330d7ec6685542067d0f4f91d78e06004ff6a193f669
c37f1dfa93bf26dd2c873af5546e6f163a3f01d02e7fdbec7738e5cfcd9eb98d
e2cd4d73d5b9ace5b7a4d75333e5d3a3e054201cff3f539ca3bc8491874b3784
inbox
All these entries are directories. I did a find . -name .dtSyncStore, which didnāt turn up anything.
Thanks for the 3.0.2 update this morning and the clear instructions. I was able to sort out (replace or delete) the ghost files before my coffee was done, turn automatic synchronization back on, and resume marveling at the powerful database in my pocket.
5 Likes
Thanks for all your efforts on this issue.
While I did have 13 ghost files (love that term, btw), most of them were some variation of āuntitled.mdā or equivalent ā so my suspicion is that they just happen to have been flagged by your ghost criteria, and arenāt the result of anything to do with upgrading from DEVONthink To Go 2.x > 3.0.
Onward and upward!
Thank you for this excellent idea. We are just exploring this option. We added an automatic verify & repair on app launch which also restores ghost files from the V2 data store if possible.
3 Likes
Assuming that would be implemented, what would happen if a user deleted DTTG 3 from the device before the legacy data had been purged-would it be orphaned on the device indefinitely?
No. The shared container only lives as long as at least one of the two apps is installed. When the last app having a grip on the shared container is deleted, iOS cleans up.
1 Like
Thanks too @Magusomo, who went hunting and posted the necessary input - I just put 1&1 together.
This is a wonderful example of what a community working together can achieve. Everybody who has patiently examined their data, posted ideas and feedback has added a piece to a puzzle which isnāt yet complete, but which is starting to turn into a picture. Iāve said it before, but it bears repeating: this forum is a nice place to be. Thank you all for your input, helping to make the software we all use better. And thank you Eric & team - itās obvious youāve been at this tirelessly; thank you for openly sharing your ideas on what is going on, opening an opportunity for us all to look at specific questions and help where we can. I wish you luck & speed going forward - youāll get the culprit in the end 
15 Likes
I have installed the DTTG update from today. I am using legacy icloud sync. The new DTTG update shows me 1 pdf in the āghost-documentā intelligent group. I see the thumbnail (0 file size). It was added on 8th January 2021 (I do not remember if I have added it on the mac or on the iphone then). I can open it on the mac (there I have no 0-file-size documents) but not on the iphone. I deleted it on the iphone. I cleaned the trash on iphone and mac. I was able to restore it from my mac. Just in case this information helps @eboehnisch. Is it safe to continue working with DTTG and DT for the moment? Or might it be that such 0-filesize documents reappear during sync process or database use?
As we are working with DEVONthink To Go 3 now since last April we deem it safe to use once the migration is completed and all ghosts of the past exorcized.
1 Like
I do not know if this mattersā¦ exactly this file has been searched for on my mac with spotlight today and was opened from spotlight thereā¦ I just try to understand why exactly this file has become a ghost file on my iphone (and none else) and why it is still existing on the mac.
(due to time pressure I am not able to read all the thread all the timeā¦ wouldnāt it be great if DT has an official thread where only the relevant information āwhat is known up to nowā is posted for all users, pinned to the top of the forumā¦ would help a lotā¦I am really afraid that I might miss anything or doing anything wrongā¦ thanks so much for having a close look at this issue and for your transparent communication)
This is, indeed, interesting information. We still believe there might be multiple factors at play here, one being the copying of the data store from V2 to V3 (even though I cannot imagine any scenario where this should fail) and synchronizing with an old sync store that contains gremlin data from the distant past. There was a bug in an older version of DEVONthink To Go that theoretically could produce such an effect but we couldnāt make it happen here on any combinations of devices. So weāre now concentrating on the more practical approach as suggested by @Blanc: Repair what we can and keep problems from spreading.
We will soon put up an official pinned post.
Hi Gang,
I just downloaded DTTG 3 and it made the Ghost smart group. It found 1 document - it was an HTML page - Not even sure what that is from - its also on my Mac. Its not an important document and may be actually totally unrelated to this issue, but I figured I would let you know.
I deleted the HTML file.
The Ghost Smart Group was NOT recreated on the Mac, I just searched for the document. Is that the expected behavior ?
Thanks