Slow Dropbox sync with large number of files

DEVONthink To Go
1

I recently built out a system that imports all my historical Twitter likes and turns them into Markdown files in a folder on my machine, and I have DEVONthink indexing this folder.

This folder has a large number of files (>10,000), all of which are very tiny (they’re Markdown files roughly the size of a tweet, give or take some extra formatting).

Problem is, DEVONthink To Go on my iPad is taking an enormous amount of time to sync these. I’ve had the iPad syncing for over 60 hours straight now and it’s still just over half way done.

Is there an alternative, faster way I can get that data moved in? For instance, is it an option to do a local sync over Ethernet while still keeping that database primarily synced over Dropbox?

I mostly ask because while I can do this on my iPad without much trouble, I don’t think my iPhone will even be able to handle syncing that.

2

Sure; use Bonjour to sync locally; the process is described in the manual. Basically on one device you allow incoming connections, and on the other you select that first device as the server - you can set it up in a matter of minutes). You can sync via multiple methods, so you can keep Dropbox and Bonjour active. Dropbox syncing is pretty unreliable for some folks (there are lots of posts on that point), so if it remains a problem, you might want to check for an alternative (iCloud, CloudMe, for example).

2 Likes
3

I am very interested in this system–would love to know more about how you did this.

4

Do you mean me? I literally simply activated several different methods of sync. So on my Mac I have set up Bonjour (Preferences/Sync/Bonjour Options) for all my databases and in addition - for two of my databases - added CloudMe (Preferences/Sync/+/Add CloudMe Sync Store). On my mobile devices I have added the same CloudMe sync stores (Settings/Sync/Locations/Edit/+ CloudMe) and the Local Network (which is the Bonjour Server, i.e. my Mac). I’ve not done anything special - it’s DEVONthink: it just works :slight_smile:

1 Like
5

Sorry–no, I meant @aharpole’s process for importing Twitter likes.

But also, thanks for the notes on syncing–I’ve been using only one sync method–but wondered about the pros/cons and feasibility of having multiple…

6

I am very interested in this system

So this system is a bit… complex right now; it uses a mixture of IFTTT, Dropbox, Hazel, and a small Rails app. You’ll also need Twitter API access which takes a few days for approval.

The thing that’s particularly nice about it is that it creates Markdown files that also have local links to the tweet and its attached images and videos, and it handles quoted tweets as well.

I’ll get this up on a GitHub repo with instructions on how to cobble together!

1 Like