Nearly my entire database of over 2000 documents is now "orphaned"

I didn’t create multiple databases with the same name on multiple computers, at least not intentionally… I created the Personal database on Mac and then synced to iOS. I have no idea why both are depicted as different versions as they used to be in sync with one another, although they would sync very slowly.

If I import from the cloud, I know that it wont have to upload, but it will have to download, right? That is the part I am worried about taking a long time.

Yes, the complete database would be downloaded. How large is the current database according to File > Database Properties?

It’s over 200 GB.

And how fast is your Internet connection?

Download speed is about 100 mbps. I am going for it. I actually already tried to download the Personal database with the Mac icon and it immediately appeared inside of DEVONthink with no data inside of it. Not sure how it got into the cloud, but I am trying the version that had the iOS icon now and it appears to be the correct size. In 15-20 minutes it has downloaded 1/12th of the data… not a bad sign.

Some weird things happened along the way, though. I was hesitant to delete the empty Personal database I attempted to download first, so I renamed it Personal - Mac Icon just to tell it apart. When I tried to import the correct version of the database, it prompted me to pick a local place to store it, as expected. When I attempted to name it Personal in the Finder, there was already a database in there called Personal in addition to the one I had just renamed Personal - Mac Icon.

I am not sure how this can be because I had already archived and relocated the local one with all the orphaned files. I decided to save the iCloud import as DEVONthink for now, and it seems to be going ok. I hope I made the right choices.

Please let me know if you think I have made any missteps and I will hopefully be able to report success on the iCloud import later today.

iCloud has been so slow with sync that I am wondering what it might look like to try syncing this database over Dropbox instead. Do you have an opinion on that? Perhaps asking the correct way to do that is subject for another thread.

As long as you archived the original database as suggested everything should be fine. By the way, which version of DEVONthink and of macOS do you use?

Dropbox isn’t very fast either but at least doesn’t depend on an asynchronous process controlled by the OS & Apple’s servers. However, one of the next releases might include a better option than iCloud or Dropbox.

3 Likes

People, we’ve got cliffhanger here… :face_with_hand_over_mouth:

I can’t wait until the first DTDC (the DEVONthink developer conference) which probably starts off with presentations in the Bill DeVille Theater at Devon Park in Devon by @eboehnisch and @BLUEFROG.

3 Likes

As long as you archived the original database as suggested everything should be fine. By the way, which version of DEVONthink and of macOS do you use?

macOS 10.15.5
DT 3.5.2

However, one of the next releases might include a better option than iCloud or Dropbox.

Very interesting! In the meantime, sounds like Dropbox is worth a try.

So now things are getting weird… The databases is appearing in DEVONthink, with correct folder structure, but a lot of the folders are greyed out and many of them have little icons to the right that look like the Finder icon. Most folders are missing the documents in them and some are half filled. Top level organizing folders like Safari, Email, and Evernote from previous imports are missing.

The downloading from iCloud progress bar in the lower left corner of the app will disappear, making me think its done, but then clicking through some of the folders to look for documents, will trigger it to start again. Is it running this whole time? Is there a way to monitor it more consistently and watch documents as they download?

One final question… I had an archive of my macOS Documents and Desktop folder in this database previously. The top level folders Desktop and Documents are now missing and all of those folders are appearing as mixed in with the folders that were actually living in the database at the root level. What data from that archive would have been kept in a cloud copy of the database. I had assumed those archives were local to the device. If possible, may it would be easier for me to disable them and restart them from scratch once the documents unique to DEVONthink are downloaded?

Thanks again.

So you’re storing the contents of the Documents and Desktop in iCloud and indexing the folders into DEVONthink?

Do you see a cloud icon next to the files in the FInder, like this?

image

Yes, I have Desktop and Documents referenced in DT and also sync them via iCloud.

As for the files being in the cloud: Some of them yes, others no. Should DT be able to search and archive the files that aren’t downloaded to the deice? Either way, I don’t remember the folders that weren’t on device appearing greyed out, empty, and with the Finder icon next to them when I first set them up.

Are all files/folders of iCloud Drive including the ones of the Desktop & Documents folder available locally?

Unfortunately, I have had DEVONthink running on my Mac, awake, all weekend, and iCloud has not finished downloading the database. I will see a download status in the lower left with a status bar, then it will disappear, leaving me to think that it is done.

Then, at random, it will re-appear, sometimes showing less or more files have been previously downloaded than it did the last time. I have no idea if it is running the entire time or not. It just shows up every now and then.

I have attached a screenshot of my Personal database. Many folders are still greyed out. I assume the ones with Finder icons were from my indexed Desktop and Documents folders? I am not sure why they lost the top level Desktop and Documents folders and are just freely floating amongst my other database files. I am also missing the Evernote subheading from my large Evernote migration. All of my notebooks from that transition are freely floating amongst the database and contain very few of the actual files. Its clear that something is working but I don’t know why the job is paused or taking so long.

I remember it taking days, if not weeks, to sync to my iOS devices over iCloud when I first set this up.

Both of my iOS devices reflect the correct organization still.

What do you recommend I do next?

Yes, items with such an items are indexed ones. You can check the path in the Info inspector.

You could either try to search for all pending items that still have to be downloaded via a toolbar search for item:pending and then try to download them via File > Download Pending Items.

Or does any of your iPhones/iPads contain a complete database including downloaded files?

Download Pending Files is greyed out. Does this mean DT thinks it is done? My iOS devices don’t have everything downloaded. My database is too big.

I still see that iCloud download status bar pop up every now and then. It reflects about 25 percent completion every time it appears though it varies.

Did you search for the pending items as suggested and select the search results?

Yes. is this correct?

That’s correct but your database doesn’t seem to contain any pending items, meaning that at least the current state of the database seems to be complete and that the grey folders are just empty.

Does a manual attempt to synchronize the database again (see File > Synchronize Database) add any new data or log any issues to Windows > Log?

Not now, but I have been getting these errors. I have also previously seen errors where it appears to be having trouble linking to my indexed Documents and Desktop files.

Do you use a Dropbox sync store too?

As an experiment, I have a small Dropbox database that I started when this issue began so that I could keep storing reference documents in the app. So far, I am having more success with the Dropbox sync, but the Evernote/Mail/Safari uploads from my Personal database had made it very huge.