Data loss when moving groups between databases

Yesterday I moved a group from one database to another and some of the moved files were lost.

Details:

  1. The moved group had one subgroup, and that subgroup had 56 markdown files.
  2. The move was carried out via drag and drop from item list to sidebar.
  3. The group hierarchy was properly reproduced in the new database and the first 46 files in both chronological and alphabetical order (the file names start with dates) were successfully transferred while the last 10 were not.
  4. The partially moved group was completely deleted from the old database, so the 10 unmoved files simply disappeared.
  5. I (immediately) verified that the files were missing from the Finder, ie. $OLDDB/Files.noindex/md/?/* had nothing and $NEWDB/Files.noindex/md/?/* had the 46 moved files.
  6. Every one of the 56 original files showed up in my system trash. This is not normal - other group moves between databases do not put anything in the system trash.
  7. Verifying the databases did not find any problems. The DT log did not report anything. I have also rebuilt the databases.
  8. After I manually restored the lost files the moved group did not sync properly. The group itself synced to DTTG but was empty there while the rest of my DT data was syncing just fine. I fixed this by deleting the database from DTTG, cleaning the database from the (iCloud) sync store, and re-synchronizing.
  9. I am a very heavy user of DT for several years and AFAIK have never lost any files before.

Irregularities:

  1. One of the 56 files was opened in another application (Emacs) at the time of the move. This was one of the files which was lost. (That was an accident, I realize that I should probably not do that.)
  2. My file names don’t translate into identical Finder/OS filenames. All 56 files had names of the form “YYMMDD: name” (without the ""s), which translates into “YYMMDD- name.md” in the OS/Finder.
  3. I (probably) moved the same group in the other direction about a year ago. A year ago group G in database A had 46 markdown files, I moved it to database B, then yesterday group G had 56 markdown files (ie. 10 new ones), I moved it back to database A, and only the original 46 files made the move.

Any ideas what the problem could have been? Does it make sense to rebuild all of my databases and clean my entire sync store?

This is indeed not the expected behaviour and so far I can’t reproduce this. Is one or are both of the databases encrypted (.dtSparse)? Was the moved group or any of its items indexed or is the destination group indexed? And finally, do you use any smart rules that might cause such unexpected results?

Thanks for the answer.

No to all three - none of my databases are encrypted, none of the moved files were indexed, no smart rules were applied. The destination group was the database itself.

I also cannot reproduce it. I can now move that group anywhere I want, in and out of other databases, with files opened in external applications, without any problems. /var/log doesn’t seem to have anything unusual. Will keep an eye on things.

Please choose Help > Report Bug while pressing the Alt modifier key and send the result to cgrunenberg - at - devon-technologies.com - maybe the logs still contain some useful hints. Thanks!

Done.

I had the same issue today. I created a new database, then dragged groups to the new database. All the groups were lost, and most of the files. Not good!

The same thing is described in the course of this thread. You will have worked out for yourself that you will need to restore the database from a backup. In the aforementioned thread, the user experiencing the problem reported back that either moving smaller batches of documents or using duplicate rather than move was successful.

As suggested in the linked thread, I think opening a support ticket is a valuable step. To my knowledge DT haven’t been able reproduce the issue, so any additional information you provide via their support system is likely to help.

1 Like

Thanks for the reply. The really weird thing is many of the missing files were in the global inbox!

Happen again today.

  1. New DB
  2. drag a Group (around 3G 3500 items) to the new DB - nothing shows in the DB but the total size if the new DB show 3G…
  3. deleted the new DB and create it again,
  4. Duplicate the same group to the new DB - parts of the data have been copied, missing around 1G. but this time with some group structure.

I’m in a situation now that I’m afraid to move a group and need to check and compare every time.

can not trust DT3 :sweat:
Need some help.

In that case the items should definitely be in the database. Is a verification of the database successful? Did you check the Trash or the Tags group? A screenshot of the main window showing the database plus a screenshot of File > Database Properties would be useful, thanks. Finally, do you use any smart rules that are triggered after moving items?

No Smart rules.
Verification was OK.
I have tried a few times with different results…none of the times ALL files and groups have copied completely.

You didn’t respond to these two things yet.
If you’re hesitant to post a screen capture here, open a support ticket and send it along.

Attach

How many items does a toolbar search for kind:any in this database find? On the screenshot a lot of items seem to be already there. Did you check the trash or the tags group of this database? A screenshot of File > Database Properties would be still appreciated.

2863

for the old group or new DB ?

Ideally both to compare the database settings too.

New DB


, AFTER duplicate of the group from the old location.

The GROUP structure did not move to the new DB

Open the smart group, select and item, and press Command-R to reveal it. Where is it located in the database?