"File not yet available" and apparent file loss

DEVONthink
1

Hi All,

I’ve recently had some major issues with my DEVONthink databases. I had been using DT for over a year with no issues like this, and though I haven’t changed how it’s setup, it’s begun to lose files.

I currently have a log message that says, “801” items left to be uploaded. When I browse through my database, there are a huge number of files that say “File not yet available”, and when the “download” button is pressed, nothing happens.

I have already verified and repaired, and also rebuilt the database in question, yet there is no change in outcome.

As it is, it appears that I have lost a huge number of essential files.

Can anyone offer assistance?

Thanks

2

Some more information might be helpful:

  • are you taking about imported or indexed files?
  • do you use DT on several machines, and if so, what method do you use to sync them?
  • which versions of macOS and DT do you employ?
3

Hi,

Thanks for the reply.

  • I’m not sure how to answer this. Can you help me understand the difference between imported and indexed?
  • Currently, I use DT on just one Mac, though I used to sync between 2 Macs with no issues. I also use DT To Go at times. I have synced through Dropbox from the beginning.
  • MacOS Ventura 13.4; DT v3.9.1
4

I just educated myself on indexing vs. importing: All my files are imported.

5

Additionally, I used this method to setup my sync initially and have not changed it.

6

The message seems to indicate sync problems. @BLUEFROG certainly knows more about that

7

Hold the Option key and choose Help > Report bug to start a support ticket.

8

Done.
Thank you.

9

I just discovered this thread today after noticing a similar problem last night. I have a single iMac that serves as my primary DEVONthink 3 machine and is left on all the time as a result. I then sync with that from my other devices: a MacBook Pro, iPhone, iPad Pro, and a variety of web browsers on other Windows machines. I went to check my notes last night on my phone and found that it said a particular note hadn’t yet been downloaded and helpfully offered me a link to click to download it. I tapped the link, but after a while the link simply came back.

I did some digging and noticed the icon indicating it had been synced but not yet downloaded, so I figured I could just sync my phone app but that didn’t fix the issue. My phone didn’t show anything more to sync, however, so I thought maybe my last changes were on a different machine or something. Only I find the iPad, MacBook pro, and even my iMac all show the same thing. If I try accessing it via the web browser, nothing at all happens–which incidentally could probably be handled better for sake of user information.

I’ve found half a dozen notes like this that I know I’ve taken and need but have essentially gone missing in action somehow. I’ve already tried verifying all the databases in question (that runs without errors) and rebuilding their indices to no avail. Should I maybe submit a ticket too? Is there any progress on this? Thanks!

10

Are you using a CloudKit sync?

11

Yes. Not the legacy one, mind you, but the latest and greatest. I flirted with other mechanisms back in the day when I was first getting started with DT, but that’s been my go-to since y’all started recommending it.

12

Given the myriad problems people have with it, we suggest other options, including local syncs. I’d say it’s the likely cause of your current issue as well.

I’d suggest restoring from your backups and putting the files back in place in the database.

13

I guess this is me learning I needed to do backups. It never occurred to me that the sync process itself could screw me over. A couple questions seem important:

  1. So what’s the “best” or at least recommended way of doing syncing if not via the CloudKit built into the app? Using just a plain iCloud folder? A DropBox folder? WebDAV on a NAS? I can do Bonjour from some of my devices but not others if that’s what you mean by “local syncs”.
  2. Is the recommended backup strategy simply copying the database files to an external source periodically or is there some better way?

Thanks in advance.

14

So what’s the “best” or at least recommended way of doing syncing if not via the CloudKit built into the app? Using just a plain iCloud folder? A DropBox folder? WebDAV on a NAS?

There is no best option. It depends on the environment and the actual needs.

Also, sync is one of the most thoroughly documented aspects of the DEVONthink suite.

The first question you need to ask yourself is, “Do I need a remote sync option?”. Consider these questions…

  • Do you have a colleague, assistant, significant other, etc. that needs frequent updates to synced data?
  • Are you using a shallow sync, i.e., Download Files: On demand in DEVONthink To Go? Otherwise, a local sync is the advocated approach.

If the answer is no to either question, a local sync on your network is suggested.

We have information available in a variety of places…

FAQ

There is a Sync section on our Questions and Answers page.

On our forums:

There is a forum post on syncing: Sync Types Explained. This is a good place to start.
There is also one specifically about Bonjour: Bonjour Simplified

Help and Manuals

Sync is covered in the Help for each application:

  • For DEVONthink, sync is covered in the Help > Documentation > In & Out > Sync, Preferences > Sync, and Troubleshooting > Sync Issues.
  • For DEVONthink To Go, sync is covered in the ? > Help > Sync your databases section.

Manuals can be downloaded here: Manuals

Support Assistant Tutorials

There are tutorials in DEVONthink’s ​Help > Tutorials​.

I can do Bonjour from some of my devices but not others if that’s what you mean by “local syncs”.

Why’s that?

Is the recommended backup strategy simply copying the database files to an external source periodically or is there some better way?

At a bare minimum, copying the databases to an external drive would work. However, it’s not going to run on a periodic basis like an actual proper backup.

As has been discussed many times, sync is ​not ​a backup, neither advertised nor advocated as such. We strongly advocate proper primary backups using TimeMachine (or similar snapshot-style backup applications) and external drives. Also, you should read the series of articles we posted:

15

Thanks for the links. I have to say this is one of the more frustrating points of using DEVONthink. I thought I understood its sync as essentially being a distributed database with no real privileged copy. But it never occurred to me that it could ever lose my data, especially when I’ve ticked the boxes to verify all uploads and such. For regular files, I use a cloud sync to keep them updated on my machines (usually Dropbox) but then also take backups to prevent my own screw-ups (e.g., deleting a file on one deletes it on all the others) even though cloud sync services usually maintain older versions. I’ve never had DT do this kind of thing before, so I didn’t realize backups were even needed.

Working through your questions, I think I need a remote sync because my computers are scattered across two homes in two different zip codes. I don’t like the Bonjour option, even though it has sometimes been my only option, because it piles up sync errors all the time about how it couldn’t maintain a connection. I’m forever clearing some sync-failed notification from my iPhone/iPad if I set them up to use Bonjour. So I was using your iCloud (CloudKit) method because I believe it had previously been recommended to me as the most reliable. So I guess I have to ask: would I be better off with Dropbox? With WebDAV? I can set up and do either of those if they’d be more reliable and lose data less frequently.

Finally, there’s the remaining question: is there any way I can find other notes/documents in this same, weird I-know-they-used-to-exist-but-they’re-lost-now-in-sync-limbo state? Perhaps a way to focus in on the metadata that makes the client use that item icon? Or a script to list them all?

16

Apple’s CloudKit, based on many (many) reports here is unreliable for many. Works for some but not all. And when it does not work there is not much you can do about it. Apple provides no “knobs to turn”. See all those posts here.

I use Bonjour very successfully but it is on the local network only. I also use Synology NAS Webdav on the local network for when the iMac is sleeping. I do not use Webdav over the internet but if you have networking skills to setup a secure connection, and maintain it, it works. If not secure you will have much trouble. There are commercial WebDav services that some use successfully. I use also DropBox for when I am out and about and i want my iPhone and Ipad to sync. Works well and reliable.

I am not going to try to bend Apple’s Cloudkit to my will when the other methods work.

Finally, it is essential that you understand how indexing files work and implications when syncing. See the DEVONthink Manual. Indexing, especially files synced by another separate system distinct from DEVONthink, adds much complexity that importing avoids.

17

Thanks for the thoughts. I’ve got the skills for secure WebDAV over an open Internet, so I think I’ll look at that and Dropbox. I don’t think the indexing thing you mentioned is relevant to me because I don’t link to anything external to DEVONthink. I always drop any documents into the store. In fact, some of the things that have gone missing are exactly that!

18

So I was using your iCloud (CloudKit) method because I believe it had previously been recommended to me as the most reliable.

It was very promising at the beginning. But over time, it became more problematic for many users and developers. However, it’s also sporadic and some people seem to have no issues (at the moment).

I’ve never had DT do this kind of thing before, so I didn’t realize backups were even needed.

Backups, especially local ones, are advocated. They are faster, generally more reliable, and of course, private.

is there any way I can find other notes/documents in this same, weird I-know-they-used-to-exist-but-they’re-lost-now-in-sync-limbo state? Perhaps a way to focus in on the metadata that makes the client use that item icon? Or a script to list them all?

A search for item:pending will show affected files but where they exist would require searching the other devices. And if you had backups, you could could restore from them to try and find them.

19

I’ve switched everything over to a WebDAV store on my Synology NAS that is HTTPS only and secured properly with a certificate. Thanks in particular, @BLUEFROG, for telling me how to find all the items I’ve lost via the “item:pending” search. As it turns out, there were only eight and they were confined to a single database. Some of them were important, others more easily replaced, so I guess as data loss goes it’s not so bad. I’ve also got a backup regimen in place, so I’m hoping I’m all set going forward. Thanks for all the advice people.

20

Wonderful and you’re welcome! :slight_smile:
Do you happen to be using a shallow sync, i.e., Download Files: On demand, in DEVONthink To Go?