Mulitple computer shared database

I realize this is a topic that has been discussed under other forum posts but none of them directly address the situation I am working with so I will pose my scenario and hope there is a workable solution or a viable recommendation. I have a client for whom I do consulting work who needs to have a single database consisting of many thousands of documents (including email) that is accessible using DTPO on two different computers (office, mobile). Only one computer will access the database at any point in time but the two machines need to stay in sync when he changes from laptop to desktop machines (which can be multiple times throughout a single day). The web server built in to DTPO is not realistic due to the port forwarding requirement and lack of assurance that when using the laptop he will always have access to a WiFi connection. Additionally, he needs to have complete access to the full database structure and documents (not just lookup searches) when on the road, even if he has no internet access. I am considering a couple different solutions and would love some feedback on their respective practicality. Because of the indexing feature in DTPO I wondered if uploading all the original documents onto his iDisk account and creating the index from the “online” files would work. Because iDisk creates a “local” copy of itself for offline use, would the DTPO database look to the local iDisk if he was offline with the laptop. iDisk would then take care of its own syncing for files that have been added when online or offline. Presumably after the initial database was built on one machine the DTPO database would need to be copied a single time from the primary computer to the secondary computer and subsequent DTPO syncing from each machine independently would incorporate new files added to the online iDisk folders. This scenario keeps all the files as independent as possible to keep iDisk syncing performance at the highest possible speed. However, the question this also raises is what about DTPO’s performance when reading and accessing files from the iDisk. If access is from the local iDisk file it should be fine. If it is from the online files I am sure it could be pretty dismal. Another iDisk option would be to keep the entire IMPORTED database online on the iDisk, but does DTPO only upload the new files added to it’s package struture or would the iDisk sync perceive that the entire package had changed and try to upload the entire thing when only a few documents had been added to it’s local copy.

If the iDisk scheme would not work at all, is there a way to easily sync the database structure (including imported and/or indexed files) between two computers to insure that they always match up using a third party application like Chronosync. Would all the internal links be correct with the documents being potentially added from two different machines into their respective local database (residing on the two different computers) and then sync’d between each other?

In either of these scenarios how are email imports from Apple Mail handled (or are they)?

Sorry for the length of this post but I am at a crossroad of whether DTPO is the way to go for this client (I hope it is because it’s functional attributes are perfect for what this client needs to do with his data). Any help or suggestions will be greatly appreciated.

Gary

There’s another alternative you haven’t mentioned, and it’s the one I would use in a situation in which I’m switching between two computers, working on the same database on each. Simply maintain the database itself on a portable FireWire drive that’s quickly unmounted/mounted for swapping between the the host computers. Small but high-capacity (hundreds of gigabytes) portable FireWire drives are fast, easily pocketable, quite reliable – and avoid all issues of synching the work done on different computers except in the case of Index-captured files that exist on one computer and not the other. Such drives are quite inexpensive. For added security, I’d keep database backups on the hard drive of each of the computers, protection against a hard drive failure on either computer or the external FireWire drive, and against loss or theft of the laptop/portable drive.

With that scenario, even Index-captured files, or Import-captured Word files in DEVONthink 1.x, can be managed reasonably well if one takes care to copy them from one computer to the other using the same relative Path, e.g., stored in folders of the same name and hierarchical structure in the Documents folder of each computer.

I’m assuming that your client has the same Apple Mail accounts available on both computers, so that should be no problem.

You indicate the client may switch computers often, sometimes more than once a day. Updating changes from one database to the other can take time. As to two-way synchronization should changes be made on both copies of the database, I’m always cautious about logical problems in such a case.

You mentioned iDisk. As you are talking about a database with “many thousands” of documents, don’t even think about running the database “in the cloud” – abysmal performance, and error-prone. Running on the “local” iDisk is OK, but there could be a very long lag before the other computer has its local copy of the database updated, so there’s a potential both for aggravation and for error. DEVONthink 2 will have a different database structure that will to some degree minimize those issues. But I would still worry about things outside my control that could happen with “cloud” computing, such as remote server problems, Internet congestion resulting in slow transfers, etc. And of course one must be online to get the local copy updated after changes on the other computer.

I’ve got a little FireWire drive myself. There have been occasions when I’m visiting a friend – Eric in Germany, for example – who has DT Pro Office on his computers, when I don’t bother to lug along my own laptop. That FireWire drive weighs just a few ounces, and holds the databases I may need. Plug 'n play. :slight_smile:

Bill,
Thanks for your informative and insightful reply. I guess “cloud computing” is running alongside the “paperless office” for the time being (albeit apps like DEVONthink are going along way toward making the latter more a reality). I do have some additional questions to ask about the external drive approach which I had considered but was trying to avoid due to the “keeping track of one more thing” problem. However, I am getting ready to leave for 4 days in the Colorado mountains with no cell phone or internet access. So I will submit my questions for your perusal next week.

Gary Sheldon (wannabe DEVONthink evangelist)

I use the “cloud” for relatively small, trivial things. One of these days, Internet download and upload speeds will go up two or more orders of magnitude, and get really usable.

Back in the early days of advertisement of online data storage services, I tried a couple of them, one free, the other for a fee. Both are long gone, along with my stored data. Yes, the trustworthiness of many online data storage services has improved. There’s a high probability that a great many of them will still be around after another year or two. I had some slide shows up on .Mac. They are gone, now. No big deal, but a reminder that things change.

With my “belt and suspenders” attitude, I want quick access to my data whenever I need it, and I like to have control of the variables that might stand in the way of access.

I’ll get one of the 3G iPhones soon, and will be pleased by data flows through MobilMe of such relatively tiny data sets as calendars and email. I hope to have useful interactions with my big DT Pro Office databases from the iPhone. But the iPhone of today doesn’t have enough horsepower to run my big reference databases independently. In the future, with more beef in the technical underpinnings of the “cloud”? Perhaps.

Right now, the full power and speed of a DT database requires local storage mounted on the host Mac, on a fast medium such as a local hard drive.

Hello Bill,
I am back in the world of technology (always good to take a break now and again). Thanks also for sharing your “belts and suspenders” attitude to maximize data safety, integrity, and accessibility. In your response you mentioned using a portable firewire drive for database storage which would allow easy plug & play transport of the DTPO files. Would, in your opinion, a USB 2.0 flash drive work acceptably from a speed standpoint (I recognize that this will be a subjective opinion)? There are currently some 16GB and 32GB options which would serve our present database needs and simplify data accessibility between the two machines (small size, no cable). Your observations about the “cloud” computing approach are well taken so I guess we will only use the iDisk as a offsite backup of the DTPO database and for simple data syncing of the iPhone.

My presumption is that the external drive approach requires the majority of the files (imported or indexed) and the DTPO database file to reside on the portable device (firewire or USB). My remaining questions derive from this approach and deal primarily with “indexed” files which reside outside of the “package” and also Apple Mail linking. If the desktop hard drive and laptop hard drive have different names, does the DTPO path link to indexed files get “confused” about where to locate the original file even if the underlying folder structures on each drive are kept identical through two way syncing? I realize that naming them identically is an option to prevent this, but it makes setting up sync scripts challenging because you cannot easily tell which machine is which. This of course presumes that some files reside only on the internal HD’s of the particular machine the external drive is currently attached to which are kept in sync independent of DTPO. This same question applies to Apple Mail. I am unclear as to how DTPO handles Apple Mail integration (i.e. does it index or import them). If they are indexed, the same question applies regarding HD naming and subsequent file location. If emails are imported, how does DTPO handle “movement” of the original email to another folder within Apple Mail? The email accounts in question are IMAP so emails originally exist on both the local machines and server (files structures are inherently identical between the two machines). Eventually (once a quarter), emails are archived off the server to the local machine folders “On My Mac” which I am presuming can be kept identical through the two-way syncing process. At this point, would the DTPO database remain properly linked to these “moved” emails, would DTPO think these are “new” not-imported emails and want to re-import them, or does DTPO only “see” the online server-based folders and not the local email folders so it would ignore the moved “archived” emails or show a “broken” link. This remains a grey area in my understanding but makes a difference on how we set all this up.

Again, I apologize for all the questions, but I have learned that proper setup at the beginning of a project of this scope this goes a long way toward preventing future “grief” based on erroneous assumptions.

Awaiting your reply,
Gary

Gary, understand that my comments are for procedures that have worked for me. There can be a lot of variables, and I haven’t tried all of them, as I prefer to keep things as simple as possible. As I use a FireWire external drive, I’ve never tried an external USB medium, especially for a large database.

For running a database on an external drive that’s frequently switched among different computers, here are some of the variables/controls:

  1. Indexed files. The database keeps track of the external files (and of external Import-captured Word files in DEVONthink 1.x) via a Path to the file – the location in which the database expects to find the file.

I prefer self-contained databases. Typically, they don’t include Index-captured files, and I minimize use of Word files. So I don’t have problems moving a database among computers if it’s Import-captured (everything copied into the database). But here’s an example I’ve created of an Index-captured file for discussion. Here’s the Path of that file:
/Users/wbdeville/Documents/Pages - MacBook Pro/HISTORY OF SCIENCE- Scientia Sinesis.pdf

What’s great about this is that the Path is relative to the user’s Documents folder. That simplifies making Paths work on multiple computers, if the external files are in the Documents folder.

  1. USB 2 external storage device. If it’s OS X/Intel Mac compatible, it should work. If a MacBook Air is used, FireWire isn’t an option.

If bus-powered for a laptop, make sure the power requirement isn’t too high.

If the USB device is solid state, choose one that has high performance and is robust for many write cycles. Running a DEVONthink Pro database will “hammer” a solid state device much more than simpler uses such as photo storage. Make sure there’s plenty of headroom for database growth.

Always leave a backup on the hard drives of the computers (Backup Archive can do that). Be prudent, especially if the data is important.

  1. Design/size databases for good performance on the weakest computer that’s used. A big database that runs quickly on a desktop with 4 GB RAM may slow down on a MacBook Air with 1 or 2 GB RAM. Think about splitting a database topically, if that’s necessary to keep performance up. RAM is more important than CPU speed.