Databases not showing in iCloud

I never experienced problems with WebDAV syncing (neither with a provider nor with my NAS).

2 Likes

What points to the issue being DEVONthinkā€™s sync being ā€œtroublesomeā€?

1 Like

FYI, I routinely use Bonjour (local sync), Webdav (Synology on local network, not from Internet since Iā€™m not travelling), and Dropbox (really only supports me when Iā€™m a road warrior). All work fine.

Yeah, me too. Sync is one of the most important features but has been a pain in the ass. I have literally burned weeks of time over the years and have tried most of the sync mechanisms, all have issues. The most reliable has been WebDAV which works most of the time. I use port forwarding to put my server on the internet.

Iā€™ve looked hard for an alternative to DT just because of sync but, frankly, Devonthink is the best thing out there. By far.

The devs read these forums and are aware of the pain, they arenā€™t in denial. If you have suggestions about how to make it more reliable and/or transparent, donā€™t be shy.

One final note. Have some sympathy with the devs. What they have achieved is quite remarkable. A multi way, multi channel, multi device sync. That is HARD to write.

If you ever have a hotel or ISP that redirects you, be careful. That is about the only failure mode I have with WebDAV. That isnā€™t the fault of Devonthink. I may experiment with a VPN to insulate DT from the redirection.

But I agree, WebDAV is the most reliable sync over the internet.

Iā€™m a tenacious man, Iā€™ll not give up!

If you carry an external hard drive, you could use a local sync store on it to sync your Mac when youā€™re traveling.

2 Likes

@sgbirtch Iā€™m curioius about your work flow and needing to sync ALL the time, e.g. from hotels or using ISPā€™s not cooperating. I of course have seen this happen, affecting not just DEVONthink but a lot of things. For example just cannot upload or download files while on some trains.

But, do you have colleagues or other computers also syncing who you need to interact with, or can you just wait for a sync when you get back to you local network (home or office)? Then do a fast Bonjour sync.

@rmschne, you asked about my work flow.

I use DT and DTTG to store home office stuff, everything really (taxes, utilities, manuals etc etc). I also like to travel quite a bit and use a motorhome for that. Everything we do and everywhere we go is recorded in DTTG (which then lost them because of campsite IP/DNS redirection grrr)

When I get back home, sync has copied everything back to an iMac Pro which backs it up using TimeMachine.

Document entry is done by Tiny Scanner when on the road and a desktop scanner when back in our brick and mortar dwelling.

DTTG also has the backup to iCloud setting enabled.

DDTG is the main program I depend on, way more than DT on the Macs. It is installed on my iPhone, iPad, and my partners iPad. Again, sync keeps them all current.

Finally, I have a MacBook Pro which is mainly used to import emails to DT. It travels with us in the Motorhome but isnā€™t used often, it gets in the way.

My vision is that all five devices will always have an up to date copy of all paperwork that runs and documents our lives.

1 Like

I do pretty much what you do, but I donā€™t use the iPad/iPhone DEVONthink ToGo as aggressively as you. I have limited ā€œdiskā€ space on those devices and prefer to use that space for music and podcasts. I donā€™t expect to use DEVONthink ToGo on the road to interact with that stuff.

I keep the ā€œstorageā€ stuff (taxes, invoices, bills, manuals) all in OSX files inside a Dropbox folder structure. When on the road and I occasionally need access to that stuff (which is not that often, e.g. when invoices paid, they are done!) I just use the Dropbox App or Files (on IOS). If I collect such files while on the road they go straight into the appropriate folder (via IOS ā€œshareā€ feature). So, that get them ā€œhomeā€ and backed up in short order assuming an online network connection. If no capable network (which happens on trains, some hotels where I no longer use their custom, coffee shops, etc.) then eventually I do find a capable network and the files move.

I have a WIP (works in progress) DEVONthink database, synced to all machines where I do note-taking, or other transient stuff. Eventually goes into where it belongs if I decide to keep. But a lot of my works in progress stuff is inside the ā€œtoolā€ where Iā€™m working on it, e.g. Scrivener, Word, Excel, or whatever. I donā€™t try to bend DEVONthink to my will to be the be-all-end-all-do-everything.

For files I have on iPad (self created or from internet) that I know I want imported into DEVONthink, I mostly use my ā€œ2DEVONthinkā€ folder method, documented here. Hard to make PDFā€™s on iPad, so I also use the ā€œshareā€ button for web pages, which I clip a Web Archive (clutter-free) into DEVONthink ToGo. That eventually syncs (right then or a little later, all controlled by Apple).

I do carry and use my MacBook on trips and I also have a small SSD drive for TimeMachine backups on the go of that device. Redundant, probably, but I have a very rigorous backup regime. Been there done that with crashes (using laptops since mid 1980ā€™s).

On the home iMac (which I call the DEVONthink ā€œserverā€) have integrated DEVONthink with those ā€œstorageā€ files by indexing them in. I reindex occasionally (someday automate that somehow as I forget). Also in the iMac DEVONthink are rules which fire ā€œon importā€ to convert Web Archive files to PDF, do OCR, classify, etc. I could put in rules to move into groups, but I donā€™t.

Since the topic is sync, I find syncing works to my satisfaction. I rely on Bonjour (local network), local webdav server (Synology), and Dropbox when on the road. Just works.

1 Like

It sounds like your use case is similar to my own although I appear to be a slightly heavier user. Mostly I just need to store documents rather than create them and sync is vital - I want all machines always up to date, as is the purpose of sync. Sync stores seem messy even if they are more reliable,

One of the things I find slightly frustrating with Devonthink is the inability to guarantee with 100% certainty the integrity of a set of files at a later time, perhaps 20 years after they were stored. Outside Devonthink I either create a tree of SHASUM files:

$ shasum * >SHASUM

And 20 years later I can verify them, although 20 years from now I am unlikely to care lying in my coffin :slight_smile:

$ shasum -c SHASUM

A reasonable but more complicated alternative is to create a git archive which uses crypto in a similar manner. Donā€™t just trust your file systems and backups, files can and do become corrupt over time, you need to be able to detect the corruption. Just by chance I happen to have a thumb in my pocket which contains a corrupt file, it is a tar file containing 10 years of emails. I only knew of the corruption because of shasum but, of course, they cannot be retrieved.

Once the file had been restored all was good and the emails became recoverable, for a couple of days. Then it went bad again. Time to toss that thumb - but it illustrates the problem.

In the context of sync (because of redirected IP/DNS or Apply failing to move files) I would like to be 100% certain that the sync succeeded. Several people have tried to exploit git for this but for reasons above my pay grade it cannot be used.

There is nothing worse than a backup which itself contains corrupted files. For the REALLY important stuff (tax returns) ā€¦ use paper :frowning:

2 Likes

I never heard of that concern re DEVONthink. Given the files are in native format mainly controlled by IOS, OSX, cloud-based file servers, telecommunications companies controlling the bits and bytes across the world, etc. ā€¦ wonder how any test means anything. Far as I know, DEVONthink not doing anything to the files unless you ask it to (OCR adding text layer, editing, etc.).

Suggest, you start a new thread to discuss and get feedback.

That is one of the ā€œworld class softwareā€ things about DT, it doesnā€™t change your files.

The shasum and git concepts allow you to store meta data about the files to guarantee end to end integrity. Any one of those technologies you mention could contain a bug and introduce a problem which you need to be able to detect before the corrupt file becomes part of your backup history. Iā€™m really talking long term here. The corrupt file I discovered contained many years of archived email.

It needs to be as provably correct as a notarised original paper document.

Maybe I expect too much ā€¦ I have worked as an engineer for one of the leading document storage companies and, at another time, one of the top fingerprint processing companies where being able to prove that prints are correct was mission critical.

So you are seeking a level of data integrity verification which exceeds that of the basic MacOS file system? How many businesses operate that that level now? For sure I know lots of hospitals, doctor offices, attorneys, local courthouses, small to medium businesses, and others who do not.

1 Like

Yes, the IT departments of those organisations are mostly run by very inexperienced engineers (50% of software engineers practicing today have less than five years and didnā€™t serve an internship) so their standards arenā€™t particularly high simply because of inexperience coupled with a lack of professional training and/or education mandated for other professions.

We are all required, by law, to retain tax records for a number of years as defined by the country where we are tax resident (or that country AND the US if you are unlucky enough to be one of the nine million American ex pats being persecuted by CBT and FATCA).

That is but one example.

I would really like to maintain those records (forms, receipts etc etc) on a computer but to do that I must be completely certain that the records in their entirety are correct and reproducible. Cryptographic document authentication is not at all difficult to implement (just use gpg) so there is no reason why we shouldnā€™t all be using it.

In general, if a paperless office cannot at least match the reliability of a filing cabinet, it shouldnā€™t be used for documents which must be stored by law.

IANAL

Perhaps you overestimate the reliability of a filing cabinet.

And underestimate the combined reliability of two or three consumer-grade digital storage systems used simultaneously in separate physical locations.

3 Likes

Sounds like a great opportunity for youā€¦ Build it and attract customers wanting it. Go for it!

1 Like

Been there done that. I worked on mission critical image storage systems back in the day (fingerprints).

Iā€™m retired now and more interested in motorhome adventures than another entrepreneurial adventure. I have sufficient technical background to make my own docs secure but Iā€™m too old to want to do another startup.