how do I sync two computers with DevonTHINK?

DEVONthink Tips
21

I’d like to announce for the first time in this thread an alpha version of DevonSync, which will allow you to sync the databases two copies of DEVONThink Pro over a network (LAN or internet). At the moment, it will check for new records in each database since last sync date and add them to the other.

Requirements:

– DEVONThink Pro 1.1beta4 (not yet released but soon) or higher
– OS X 10.4.x

Limitations:

– A know OS X keychain bug makes it necessary NOT to store eppc: passwords on keychain or else get crashes ; means typing passwords a lot.

I will eventually be releasing as unrestricted shareware. Right now I need some testers.

Anyone who is interested please email woodenbrain ----=== at -======----- kagi ----===== dot ===== ------ com

22

Cute joke. :slight_smile:

But I wouldn’t touch software that claimed to be able to do two-way synchronization between DT Pro databases with the proverbial 10-foot pole – although I’ll try out one suggested as experimental by Christian, on copies of my databases.

Yes, it may be possible. But the monolithic structure of the current database makes two-way synchronization a non-trivial problem. It’s easier to synchronize files stored in the database Files folder than in the database body. And it’s easier to handle file changes than database organizational changes.

DT Pro 2.0 will introduce a new database structure that will make many current synchronization problems go away, or become easier to resolve.

But I do in fact manually handle movement of DT Pro databases back and forth between my iMac and TiBook, when I take trips. Here’s how:

[1] The easy scenario. Copy the database package from the iMac to the TiBook. Or copy the database package from the iMac to a portable FireWire drive that I’ll hook up to the TiBook, and on which I’ll actually open and run the database (my current practice). Make content additions, deletions and edits as desired. On return from the trip, simply replace the database copy on the iMac with the database on the TiBook or the portable drive.

[2] A more complex scenario. I’ve transferred the database to the TiBook as above and made changes. Here’s a complication. Changes are also being made to the database on the iMac at the same time, either by me or by someone else. Here’s how I end up with a database that contains the new contents added to both databases. I choose the database on which the smallest number of changes was made, open Tools > History and select all the items that were added during the time interval of multiple use. In the Finder, I create a new target folder to which I will export those added documents. Back in DT Pro, I select File > Export > Files & Folders for the selected History items, and send them to my target folder. Assume this is the database on the iMac. Now I can replace it with the database copied over from the TiBook (or FireWire drive), and then do File > Import > Files & Folders into that database. Result: I’ve got a new database that contains all the added content made on both computers during a specified time interval.

Discussion: There could be other changes that couldn’t be handled by approach [2]. Example: I totally reorganize the group structure and move contents around between groups in one database. Unless this is the database that I keep and import added content to, I would lose all that work invested in changing organization.

When version 2.0 is released, the database structure will be modified and many synchronization issues will be easier to handle. But that isn’t to say that I would immediately trust off-the-shelf two-way synchronization software to manage my database movements and consolidations. Misplaced trust can be dangerous. :slight_smile:

23

Bill,

That’s unnecessarily mean and invalidating. I’ve been interested in this since I bought DEVONThink and as Christian and Eric both know, I’ve been scripting for it. You know me as well under my real name. This is in fact the experimental one that Christian mentioned to you, and he already has 0.2 alpha.

Option 1 assumes you have a portable drive, or carry your laptop back and forth. It also assumes that your work with DEVONThink is of such high priority that you want to begin and end your work day by copying databases from one computer to the other and repeating at home and/or keeping extact track of which database is the newest, AND never modifying both databases before undertaking this laborious process. In my case, as I suspect with others, which is why there is some clamour about it, have a computer at home and one at work, and an internet connection whose upload bandwidth over cable is capped at about 50K/second. With a 400 meg + database and growing, it would take over 45 minutes to copy the thing from home to work given the limitations of my upload bandwidth. And that’s assuming my upload bandwidth isn’t being used for anything else. I’ve explained this to Eric and to Christian and I believe to you as well. So without meaning to be as rude as you were about the 10 foot pole, I sincerely wish you wouldn’t just ape that line without acknowledging the limitations.

Option 2 is in fact more complex. In part this is what DevonSync handles. In fact it handles it better than what you are suggesting, because it does maintain the group hierarchy. The only drawback is you will end up with two databases that are not identical because mainly the internal ID of the records added will be different.

24

Sorry, didn’t mean to be rude, but I read the email address at the end of the post as a deliberately fake one. So I thought you were joking.

Further complicating the matter was that I read your post before seeing Christian’s email about the script. So I didn’t connect the experimental script with your post. Mea culpa.

You are certainly correct in that there would be great interest in synchronization of databases across a network.

I think that such synchronization will be easier to handle with the upcoming version 2 database structure. But if you have made progress with handling changes in the database hierarchy in addition to incrementally updating document content changes in the current monolithic database, that’s a real accomplishment and could become a very useful product.

Yes, I did go through a period of transporting my databases via a FireWire drive back and forth from office to home, but it only took 5 minutes at each end. Granted, there was no possibility of a change being made except on one computer, so it was easy to keep track of things. I didn’t have broadband at home at the time, so a network connection was out of the question, anyway. Now I’ve got broadband and a WiFi network.

I’ve got a real-world situation on which I’ll try out the script. I’ve got a database on my TiBook (copied from my desktop computer) that contains new content and a few new groups, but the original database on my desktop computer has had substantial changes in group organization since the copy was made, as well as several thousand added documents.

But with my “belt and suspenders” attitude, I’ll work with copies of both databases. :slight_smile:

25

Bill,

No problem, apology gladly accepted. I just wanted to emphasize I’m serious about this because it has long been a real need as you know.

DevonSync at least in its current form will not handle the situation you describe very well. I would recommend for databases that are very different that you use method 1 as you described the first time. Then incremental changes that are in the nature of adding documents over a few days can be handled by DevonSync.

I am up to 0.4a right now, and will gladly send you a copy by e-mail if you like.

26

Thanks. I’m sending you the email right now!

27

Hi Wooden Brain,

I would be very interested especially if it were possible to do the synchronisation via .Mac possibly by creating some kind of synchronisation file in a .Mac directory.

Bill, you are without doubt one of the most knowledgeable people around when it comes to DT but sometimes I feel that you dismiss other people’s ideas and approaches too quickly. In this case the issue is a programming related one and with respect comments on its validity should come from programmers such as Christian. Sure synchronisation of databases is a non-trivial programming problem but there have been solutions to this issue for a very long time now. Many people use iSync to synchronise phones, palms, macs etc. every day without any major difficulties so the problem is clearly solvable.

Wooden Brain, I would be interested in beta testing DevonSync but unfortunately I’m not on the current DT beta programme and if I unuderstood your post correctly the current beta version of CT is required?

28

DevonSync goes directly over the network between 2 computers (LAN or internet) so there’s no need for .Mac. I wish iSync did the same thing actually, and there’s no good reason why it shouldn’t, except for the fact that Apple wants to sell it’s .Mac subscriptions. (I’m a sucker there.)

With regards to testing, if you e-mail me, I will (by permission) give you Christian’s e-mail, and if you contact him he will add you to the beta list if you mention you are testing DevonSync too.

29

Peace, folks. Bill responds to hundreds of e-mails every week. His information is good and his patience legendary. In this case, messages arrived out of synch, as he explained. Let’s not pile on our evangelist!

30

This sounds very promising. Thanks for taking this on.

:slight_smile:

cheers

My hat is off to Bill and other DTers who continually handle our sometimes frustrating issues with grace.

31

I need to synchronise a DT database between two remote computers rather than directly over a LAN. Does DevonSync require two copies of the database running simultaneously to achieve it’s ends? I think what would work for me is to be able to upload a “difference” file to my .Mac account and then when I launch DT on another computer have it check automatically for this file and apply it to the just opened version. Ideally as well when quitting DT it would create the difference file in my .Mac account automatically. Obviously it would need to be a bit more sophisticated than this since if I only worked on one machine for a while it would need to either create multiple “difference” files or it would need to append to the original one. I suspect that some of this may already be possible using built in scripts and automation.

32

You can network two computers over the internet and it does not have to be on a LAN. you just need the external IP addresses of both computers. So there is no need for .Mac or any intervening server, as long as both computers have internet access. DevonSync requires that both computers are on, both are running DevonThink Pro, and both have internet connections. Is that in of itself a problem for you? In any case why don’t you just e-mail me and I’ll send you an alpha. You also need to get on DTP’s beta test group to get version 1.1beta 4 or higher of DTP on both computers.

33

Hi Wooden Brain,

Not sure how practical your solution will be for me without establishing a VPN or using ssh tunnelling etc. My home computer has a broadband internet connection but the IP assignment is handled by the ADSL box. Not sure at this stage if I can determine the IP is use or over what interval it is static although I’ll check on this further. The real problem I suspect is my work machine which is on a LAN attached to the net by a router and a firewall. It currently doesn’t use a dedicated IP to connect to the Net and I will have to configure the router and firewall to allow for this. I can certainly do this if necessary.

Could you provide a little bit more information about how you see the two computers being connected. For example were envisoning using say ARA, VNC or Timbuktu?

34

It’s best if you just e-mail me and I’ll send you the latest alpha, and then you’ll have a better idea what the setup looks like. There is some config info in the application already, but your posts show me I could put some more details in.

I’m not a network administrator. But you’re making it sound more complicated than I suspect it is. In a work network situation like you describe, incoming traffic passes through a firewall (which might just be his/her computer) controlled by your network admin. He can set what traffic coming over what ports go through to what machines.

DevonSync uses two protocols, Apple File Sharing (a.k.a Personal File Sharing) and Remote Apple Events. The first, which is addressed in a URL as eppc://, uses port 3031.

So all your network admin has to do is allow ports 548, 427, and 3031 to go through, either to your computer or all computers. If you DID NOT have a network firewall, you would so the same thing in the Sharing Preference pane, ie, turn ON the OS X firewall but allow the protocols you want, in this case Personal File Sharing and Remote Apple Events.

35

DevonSync 0.5a is available, and DTP 1.1b4 is out as well. So anyone who wants to test this can now. So far I’ve sent it out to a few people but heard nothing back after a few days. So please volunteer to help test. Thanks.

36

I’ve sent some feedback regarding my experiences with Devonsync. If you don’t get it, post here and I’ll resend the emails.

37

Will this nirvanaous activity eventually be possible with Devonsync?

38

¿Que? If by your neologism you mean will you be able to sync with devonsync, the answer is yes, with some limitations:

  1. design limitations: it’s an add sync only — will add to one mac what has been added (by you) on the other, in the right place.

  2. network limitations: getting both Finders and both copies of DevonThink to communicate is tricky. DS tries to make it as transparent as possible, but the user has to know something about his/her network setup, and the network gods have to be in a good mood.

  3. development and testing limitations. Right now it’s in alpha stage. On my end it is working functionally with 2 databases that are pretty close in structure making incremental changes. I’ve heard back from one tester though, who has had lots of errors and issues to report.

So whether this gets out the door or not depends on good testers, good feedback, and the availibility of my time to hammer out issues that may crop up in usage that I don’t expect.

39

There’s hope:

MySync - Mac to Mac syncing without .Mac

Although it’s already been getting positive feedback I’m not enough of a lab rat to try it before it’s out of beta since it could be tricky to undo damage.

However, I may be interesting in beta testing your DEVONsync (preferring DEVONapp-style capitalization for consistency’s sake :slight_smile:) utility since damage recovery could be as simple as reverting to pre-testing databases. But first I need to submit some overdue bug reports to DEVON regarding DTP’s inability to cleanly rebuild of my primary database and a few other issues.

40

  1. the mac - to - mac synching is not the same thing – the point of writing DS is to avoid the huge transfers of the entire database over a network when there have only been a few addiitons on either side. there are literally dozens of network synching options that don’t use .mac if you want to transfer whole files.

  2. sorry about the caps. i may have to fix that. even though I have to say i never much liked the caps on DEVON.

  3. if you want to test, please e-mail me and i’ll add you to the group. i’m having a lot of trouble getting any feedback. (one good feedback email, you know who you are). anyway, i’m going to rebuild it as universal and get it out the door soon.