Mac on demand sync Workaround (Tested)

Hi everyone, first post here.

I’m currently testing DT and want to share an eureka moment I had, maybe could help someone. I want to one-way sync a huge archive index-only database on my desktop, to a low storage space MBP. But DT currently does not support on-demand downloads on Mac (DTTG does though).

Scenario
Old 2011 iMac (High Sierra):

  • Main drive 512gb SSD with Dropbox account.
  • External 6000gb spinning HD with an Archive directory (/Volumes/exthd/Archive/) with some big folders (50-250gb each). Folder contents don’t change.

2017 MBP (Big Sur):

  • Main drive 512gb SSD.

Goal
MBP: To be able to search Archive directory; and if something is found, download it “on demand”.

Problems

  • There is no enough storage space on iMac to put the Archive directory on my Dropbox.
  • There is no enough storage space on MBP to sync full Archive database.

Solution
iMac:

  • Move Dropbox account from the main drive to the external HD; and move the Archive directory to it (/Volumes/exthd/Dropbox/Archive/)
  • Sync Dropbox (took a while)
  • Index each folder of the Archive directory into DT databases
  • Set up Archive directory to “on demand sync” in dropbox
  • Move Dropbox account back to main drive

MBP:

  • Set up Dropbox account with “on demand” setting
  • Sync Dropbox
  • In terminal:
# sudo mkdir /Volumes/exthd
# sudo ln -s ~/Dropbox /Volumes/exthd/Dropbox
  • Bonjour sync DT Databases

Result

  • Now the DT Databases are indexes that point to Dropbox on-demand files

Comments

  • Probably not an elegant solution but worked for me.
  • Any other ideas out there? This solution looks fragile. But gets the job done.

Keywords
LOW DISC SPACE
LOW STORAGE
ON DEMAND SYNC

Links

On-Demand Sync on Mac? - DEVONthink - DEVONtechnologies Community

2 Likes

If it’s just an archive and the contents of the archive are indexed & only modified on one computer, then there’s another possibility:

  1. Index the archive on the main computer using a path that is available on all computers (usually a folder in your home directory or its subdirectories)

  2. Create a new sync store and upload the indexed archive

  3. On the other computer(s) add the same sync store but (and that is important!) disable the option to synchronize the contents of indexed documents

  4. Import the archive database from this sync store to the other computer(s)

The search index & metadata are now available on these computers but the files aren’t. However, you can download them on demand, e.g. via the preview pane or via the File menu.

The only limitations are currently that you can modify the files of the archive only on the main computer and that you can’t remove local copies anymore on the other computers after downloading them. Depending on the available space and how often files are downloaded this might be nonetheless an option.

2 Likes

Thank you!

If I understand correctly this means that on the target machine (MBP in my case) I can:

  • Sync the index without document contents, even though the sync location is configured with full sync on the source machine (iMac in my case).
  • When I try to preview content, DT will connect to the sync location and download the file into the indexed path in the target machine

This looks like a proper solution. But in my case, the problem with this approach is that your first bullet does not apply:

Index the archive on the main computer using a path that is available on all computers (usually a folder in your home directory or its subdirectories)

Because the archive does not fit on my source machine’s main HD, and DT does not follow symlinks AFAIK.

Thank you for your thoughts on this!

It’s a shame people have to come up with elaborate workarounds for a feature that could be built in and allow users to choose for themselves whether to download all files or download on demand.

I think some of the assumptions designers make about the limited storage and power of mobile platforms are no longer true. The newest 12.5 iPad, for instance, has a capacity of 1T storage and uses the same M1 chip found in the silicon Macs.

The option to “download on demand” on synced Macs as well as mobile devices would give customers the choice to manage limited storage space as they choose, and be able to use the program on Macs with limited space where they cannot now use it.

I would disagree with you from a fiscal perspective…

Note everyone can just drop between $1800 and $2000 on an iPad. In fact, the 128GB model is still $1100.

And you can get an M1 Air with 256GB internal for $1000 (and it’s a little monster of a machine :heart: ) that’s as light as the new iPad Pro.

Even decked out with 16GB RAM and a 1TB internal, it’s still $400 cheaper than the iPad Pro.

Regardless of your personal fiscal choices or arguments, the fact is mobile devices are becoming much more capable and offer much more storage space, and the reality is that people are buying and using the new M1 iPads.

It is also a fact that many Mac users also obviously have a need and desire for the ability to choose whether to download all files or download on demand. It would be appreciated if the DevonThink developers would seriously consider adding this option, instead of just dismissing it based on questionable assumptions.

It’s actually planned for future releases (but not in the near future).

3 Likes

That would be great, thank you! The sooner the better … :smile:

That’s true for all the features on our endless to-do list :grinning:

6 Likes

There is a song like :musical_score:“I have 100 bugs, I’ve resolved one, now I have 126 bugs”. :musical_score:

And another: :musical_note:“I have a list of 100 pending new features, I implement one, now I have 45564353 in the list of pending new featureeeees” :musical_note:

:innocent: :sweat_smile: :joy: :rofl:

3 Likes

Especially the second one sounds very familiar :laughing:

2 Likes

Well no, at 2.8# it’s nearly twice as heavy as a 12.9” iPad Pro. In any event, there are other factors that make the iPad Pro a better choice than a Mac other than cost considerations. The pencil, forward-facing camera, and a cellular option that doesn’t require you to hot spot your Mac to a phone when away from WiFi are features that are important to some users.

Count me as another user hoping to see this feature sooner rather than later.

I’ve been been struggling to solve a certain file sharing situation and have already spent roughly two full days of time trying different alternatives. If DT on the Mac had the same capability as DTTG does with respect to the “download files: always” setting, then my problem would have been solved already.

Specifically, what are you struggling with? What are you trying to do?

I’ve never noticed a file not being available when wanted as I’ve always sort of assumed based on what seemed to be happening that files were always downloaded on sync… Maybe not?

As others in this thread discussed, the description above is not how an indexed group works in DT on macOS (although it is, in effect, how it can be made to work on DEVONthink To Go on iOS).

Suppose you have a network-shared drive on a local network, and that this drive is not accessible outside the network. Suppose you want to have (in DT) an indexed group over something stored on this network-shared drive. You mount it normally on your macOS computer via SMB, and index it, and all is fine as long as the computer is on the local network. But if you want to do the same on a laptop that you then take outside of that local network, then you discover that outside of the network (where the network-shared drive is not accessible) the files in DT’s indexed folder are also not accessible – in DT’ indexed folder view, you can see they exist, you can list them, search, etc., but not actually view the content of the files.

The way most people solve this is by putting the files & folders in question in a location that is cloud-sync’ed by something external to DT on the Mac. This could be (e.g.) Google Drive, Dropbox, iCloud Drive, OneDrive, etc., or one’s own NAS/DAV/etc server. In some other cases, another application might have its own sync mechanism and provide (in effect) a cloud-shared folder of its own. For example, I use Zotero for reference management and I index the Zotero folder in DEVONthink, and it works beautifully on my desktop and my laptop and my ipad – because Zotero takes care of synchronizing the files & folders on the desktop and laptop even if they’re on different networks, and on the ipad (unlike DT on the Mac), DTTG can download and store the files from the sync store at sync time.

The TL;DR is that you have to put the indexed content in a location that is accessible from all computers where you want to run DT with an indexed group over that content.

In my case, I have constraints that prevent doing this in a simple way. I did solve my particular set of constraints, but with a lot more management and care, and the result is still fragile. The only reason I even have a complicated set of constraints is that I’m such a die-hard DEVONthink user at this point that I really, really want to have the content in DEVONthink on a laptop that I can take anywhere. And if DT on the Mac had a symmetric option to DTTG’s “download always” feature, then my (admittedly unique) problem would not exist.

(To be clear: I’m not trying to blame DEVONthink’s developers for causing these difficulties :slight_smile: The software works well and was designed to meet certain goals and use-case scenarios. One can’t blame the developers for being unable to satisfy every snowflake that comes along.)

I was thinking it… and you said it!
KIDDING! :wink:

But thanks for the laugh, @mhucka :slight_smile:

1 Like

Retaking this, one solution is what I did: buy Macs with enough disk space (1 TB in my case) to keep all files, but I vote for on-demand as well.

There is a hidden feature in DT that perhaps ease the scenario presented above: “DisableAutomaticUpdatingOfIndexedItems”. It is documented in Appendix → Hidden preferences.