Feature request: Use iCloud files instead of indexed ones

DEVONthink To Go Feedback
1

Lets suppose a Database with mixed indexed and non indexed files. Indexed files are all located in iCloud Drive…

In macOS those indexed files must be local, but in DTTG they are stored in Sync DB. However, if those files aren’t (because sync database has been marked to not to stored indexed items), and as they are in iCloud Drive with full access from DT as iCloud Drive is integrated into iOS…

What about being able to get the files from iCloud Drive instead of sync database?

What I mean is, having in DTTG download files On Demand, when the file is not local, get it from iCloud Drive instead of sync database. I think this will end with less duplicated (a problem I’ve stopped having since I’m using a beta version), smaller sync databases, less space in DTTG, and a step less in DT “magic sync” (have a file in iCloud Drive, sync it via DT into DTTG, modify in DTTG, sync to DT one more time, and DT saves the modified file in iCloud Drive).

This way, those that index files from iCloud Drive won’t need to go back to macOS DT in order to have synchronized those files.

(And of course, on next future updates, this could be added to Dropbox as well)…

1 Like
2

I recently found this article, which gives some insight into what is happening when iCloud sync takes place: https://eclecticlight.co/2020/01/21/testing-icloud-using-cirrus/. All I can say is that it looks even more complex than I thought it was.

3

Interesting. Some time ago, I think it was in High Sierra I had a Mac what took a lot of time to show the iCloud folders and I installed this Cirrus (or another similar tool) that showed what was happening in the background, and as it appears that nothing was happened, internally was working “fast and furious” trying to get the files.

However, cloud sync is easiest that is seems. When you modify a file or create a new one, it is uploaded, verified the upload and an entry added to the table of files, ordered by upload/modification time. (However, seems iCloud downloads one more time the file to check it is ok and to replace the uploaded one). If there is enabled the “placeholder” stuff, the file is replaces by a placeholder containing the metadata needed to get back the file. (In iCloud Drive, it is a .iCloud text file containing an UUID and nothing visible more).

When a file is modified externally, a notification is sent to all active machines and then the file is checked and downloaded (placeholder or file itself).

What could be wrong? Local file is locked (you get a duplicate), local file has a timestamp too near to the remote one (you get a duplicate). Clocks in devices could be not in sync, then iCloud/Dropbox/whatever needs to have an internal timestamp stuff, perhaps the most complex thing to implement here.

If your device has been switched off, or has not received the notification, the only things that local machine needs to get is the list of the last modified files since last connection. And this is the reason once you switch on a device, ten seconds after you get (from Dropbox and OneDrive, but not from iCloud Drive due “dark” Apple way to do the things) a notification that X files have been modified/added, basically the time to check those files against local filesystem.

When a device is not connected, and two same files have been modified in different machines, you get duplicates as well.

What is very complex is the “placeholder” technology, that must be implemented at low level operative system driver, as it finally has been done both by Microsoft and Apple. And in both cases, first incarnation of this technology failed miserably (in Windows 8 for Windows, and in previous versions in macOS). And it seems both companies have shared their how-know to make this possible, as both Windows/macOS got more or less at the same time the now stable “placeholder” tech, I think it is the same as OneDrive and iCloud both works fine in Windows and macOS and icons, menus and other things are completely similar. Most critical thing here is have a hook to pass to the cloud controller the file that is being opened to download it.

4

I expect the official response to this from the devs is that iCloud Drive is not the same as iCloud sync, so DEVONthink to Go is not going to be able to flip between documents synced to the database via iCloud and indexed files on iCloud Drive. If DEVONthink to Go could directly access indexed documents on iCloud Drive, I don’t see why there would even be a reason to have sync databases (assuming that the user is fine with indexing all documents).

1 Like
5

Your comment has two parts. First one is the sync database, that does not need to be iCloud sync, but Dropbox or WebDAV (my case). Desktop DT uploads the indexed files into that DB.

What I mean is if the indexed file is stored in iCloud Drive, DTTG could directly access to it as any other app is able to access to iCloud Drive… For example PDF Expert or PDF Viewer from PSPDFkit. They have full access to user files.

And the second is sync database is still necessary to sync metadata and maintain database structure.

6

But DEVONthink doesn’t know and doesn’t care if your files are in iCloud Drive. Indexing documents that are also synced to iCloud Drive and indexing documents that are also synced to Dropbox, etc are all treated the same in DEVONthink to Go. I can’t visualize how a developer can code an iOS app to go search all the connected cloud services to see if there are documents that mirror what is in the synced database and then make them accessible in DEVONthink to Go.

7

DTTG knows if a file is stored in iCloud Drive or in whatever place it is stored, as it has its path, and the path is the same in both macOS and iOS:

~//Library/Mobile\Documents/com~apple~CloudDocs/

However, if that is not the case, both have the relative iCloud Drive access path.

8

At the end I’ve found a very serious issue: in iOS, an application cannot access outside its own iCloud Drive folder and the only way to do it is open the iOS integrated Files pickup control…

However, perhaps there could be some “exception” if asking Apple (Years ago I got an exception from Microsoft Windows Phone in order to implement some VOIP stuff I was doing - Apple should have similar rules-exception).