Chronosync and Devonthink?

Any of you using Chronosync to sync databases between macs? They have a dissect packages option, seems simpler than the syncstore option provided by Devonthink.

I have used ChronoSync for years to sync databases (and other material) between machines on my LAN. It has never failed me, is reliable, and easy to configure and administer. On one machine I have had ChronoSync sync and clone jobs running automatically several times daily for years, unattended. I don’t use “dissect packages” mainly because I’d rather not troubleshoot things like that – if bits and pieces of a database go missing it is very difficult, time consuming, and therefore costly (unbillable time) to sort the problem, regardless of logs or any other aid.

(That daunting task, not easily possible with DEVON Sync, is the main reason I’ll never use DEVONthink Sync. In fairness, DEVON Sync is not for backups, and should never be used for that purpose. Whereas, Chrono Sync and it’s competitors were designed for backing up data.)

In addition to the difficulty of error-shooting DEVONthink databases, when I evaluated ChronoSync’s dissect packages I found that if not configured precisely date/time stamps in packages can introduce subtle errors in databases that cannot be repaired with tools provided by DEVON or others. So, it’s just easier to copy entire databases among machines. If ChronoSync fails it is always because a target machine or drive went offline, and all I have to do is start over. For me, none if this is time consuming because I’ve used ChronoSync’s scheduler to run these jobs when I’m out of the office or at night, and I’ve worked out a method of tracking which machine has the current copy of data, to avoid using the wrong instance of a database.

There are many fine competitors of ChronoSync, so it is not the only alternative.

Korm:

So you would actually recommend Chronosync instead of the built in sync option in DTPO?

Do you use bidirectional sync or the backup (copy from a to b) option in Chronosync?

I recommend writing down, prioritizing, and evaluating your needs and comparing them to the available options, which you test and judge based on your needs. Which is how and why I chose ChronoSync. I needed a safe, reliable, proven trustworthy backup solution for the data on which my income relies. DEVON Sync did not exist when I chose ChronoSync. When it became available to test I evaluated it in the same manner; it did not meet my needs and performance requirements; so I don’t use it.

Your needs will vary from mine.

Depends on circumstances. If I’m moving database between a desktop and a laptop, I usually use the backup option and have an A → B job and a separate A ← B job. Do the first job before I leave the office with the laptop and the second job when I return. On the other hand, if neither machine ever leaves the office, I usually run bidirectional syncs scheduled during the night.

Korm:

Appreciate your insightful replies.

Maybe I missed the point here - Is it true that you don´t use the dissect packages option in Chronosync? But how can you be sure that everything is synced properly?

I don’t use dissect packages. I close databases, sync the entire database (I.e., the entire package as a whole). I let CS verify the copy.

Others on the forum have had no problem with dissect packages – search and you’ll find those threads. I’m just very conservative when it comes to these data.

Korm, is using Chronosync (which is what I do as well) still your preferred method of keeping databases in sync between two Macs? Or have you switched over to the in-built sync feature of 2.5?

Korm, is using Chronosync (which is what I do as well) still your preferred method of keeping databases in sync between two Macs? Or have you switched over to the in-built sync feature of 2.5?

I have no intention to switch to DEVON sync. I tested DEVON sync extensively during the beta period, evaluated the feature alongside my specific requirements (which include backup**), and found that the status quo using ChronoSync met my needs.

Other readers have different needs and will reach their own conclusions.


**[size=85]it is worth noting: DEVON Tech has frequently stated that DEVON sync is not intended for backup. I have to have a backup solution. Econ explicitly advertises ChronoSync for backup; therefore it met my requirements. [/size]

I’m feeling the same way.

Just experimented today with syncing a database using the new feature, but it crashed my DTPO.

Chronoysnc it is, then.

I have used ChronoSync for many years to sync a database between my work mac and my home mac, using an USB stick. This works reasonably well, if you are disciplined enough. I am often chaotic and that caused problems. I hope DevonThink Sync will do a better job, but have not tested it well enough to give a verdict.

ChronoSync works well if you always sync before opening a database and again after closing a database and never when the database is open. So you arrive at your work, sync, open DEVONThink, work within DEVONThink, close DEVONThink, sync, go home, sync, open DEVONThink, work, close DEVONThink, sync etc.

One problem is the time it takes to sync a database. In my case syncing with dissect packages could take half an hour or more, which is a long wait if a colleague offers a ride and wants to go now.

Another problem is haste and absent mindedness. For example, in the afternoon I had often forgotten that in the morning I hadn’t had the opportunity for syncing and in such case I often opened the database before I became aware that I hadn’t synced yet.

Originally I synced using dissect packages because it is much faster. This resulted in inconsistencies and other verification errors after syncing every time I opened the database before syncing it.

I then deselected dissect packages. In case of deviation from the discipline this results in synchronization errors rather than verification errors, which is much better because synchronization errors, unlike verification errors, can’t cause corruption and unlike verification errors (which only show up when you actually do a verification) they always result in an error message.

However, as my database grew (the one I am talking about is currently about 10 Gb) synchronization with dissect packages deselected started to take hours, so I switched back to dissect packages.

I expect that these problems will not occur with DEVONThink Sync, because, if I have understood the scanty :cry: information well enough (and I am not sure that I have) it does data syncing rather than file syncing. This would mean that it won’t cause problems if I add a file (W) at work and another one (H) at home without syncing in between. At the next sync at home H will be copied to the sync store. When I then sync at work H will be added to the database at work and (unlike when syncing with ChronoSync) that database will be “aware” of what happens, and it will update its administration.(*) At the same sync occasion W will be added to the store. At the next sync at home W will be added to and registered in that database. No problems at all.

It would also mean that DevonThink Sync is faster than ChronoSync.

(*) If you use ChronoSync with dissect packages, at this point H will be transferred to your work database and W will remain in that database. However, as the transfer occurs without DEVONThink being aware of it, the files that describe the database’s organization will know only about one of them (H or W), or, perhaps, some of the administrative files will only know about H while others will only know about W. Hence, the verification errors.

I have found DEVONthink sync no faster, and in most cases significantly longer, than other methods of sync – including sneaker net. If a file in DEVONthink is changed, then it will be copied to the sync store by DEVONthink sync. DEVONthink does not decompose your files to smaller chunks of data. I’m not sure what the quoted comment means – but if it is meant suggest that DEVONthink is looking for bit- or byte-level changes in your files and only copying those atoms, then that is not what happens with DEVONthink sync. It is very easy to examine what is going on by experimenting with a small test database.

Actually, if you do not dissect packages your sync is more accurate than if you do use it!

A DevonThink database is what is called a bundle. A bundle is technically a folder, but it is a folder that should be treated as a file. ChronoSync treats bundles as files, except when you enable ‘dissect packages’ in that case it treats them as folders.

The difference is best explained by means of a simple example. Suppose the database folder contains one administration file (A) and two documents (B and C). Source and target are in sync. Now you add a document D to the source (meaning that A is changed).

So the situation is this:

Source [A’,B,C,D]

Target [A,B,C]

If dissect packages is not checked ChronoSync will replace [A,B,C] in the target by [A’,B,C,D].

If dissect packages is selected ChronoSync will replace A in the target by A’ and will add D to the target.

So, sync with dissect packages is less work and much faster.

Now, consider what happens when source and target are both modified. For example you have added D to the source and E to the target. In his case at the moment of syncing the situation is:

Source [A’,B,C,D]

Target [A",B,C,E]

When the database is treated as a file (that is when you haven’t checked dissect packages), depending on the direction of the sync and your conflict solution settings you will get a synchronization error, [A",B,C,E] in the target will be replaced by [A’,B,C,D], or [A’,B,C,D] in the source will be replaced by [A", B,C,E]. In all cases the integrity of your database is maintained.

What happens when dissect packages is checked depends on many things, including the sync direction, your conflict solution settings and your actual decisions in case of conflicts. However, whatever the settings are and whatever you decisions the integrity of your database will break (that is, you will have validation errors).

Suppose for example that you have a bidirectional sync, in that case E is copied to the source, D is copied to the target and A’ and A" are in conflict. Suppose you resolve this conflict by choosing ‘source to target’. In this case both source and target become [A’, B, C, D, E]. However, because A’ doesn’t know about E there is a validation error.

Suppose your direction is from source to target. In this case D is added to the target and A’’ in the target is replaced by A’. In that case the source database remains valid [A’, B, C, D] but the target [A’,B,C,D,E] not (A’ doesn’t know about E).

Suppose you now modify D in the target and sync from target [A’“, B, C,D’,E] to source [A’, B, C, D]. Now E is added to the source, D in the source is replaced by D’ and A creates a conflict. If you resolve this conflict by favoring A’” the source [A’“, B, C, D’] becomes invalid because A”" doesn’t know about E; if you resolve the conflict by favoring A’ the source [A’, B, C, D’] becomes invalid because A’ doesn’t know about E and thinks that D’ is D; if you resolve the conflict by keeping both the source [A’,A’“,B,C,D’,E] becomes invalid because of inconsistencies (according to A’ there is no E and there is a D, but no D’, according to A’”" there is an E and an D’ but no D)

That’s why syncing without dissecting packages is easier and more secure than syncing with dissecting.

Hope this helps.

Now I have tried to sync the Devonthink database (xxxx.dtBase2) from my Macbook to my iMac through WiFi using chronosync on my Imac and Chronoagent on my Macbook. However, I am unable to choose the dtBase2 file on my Macbook (it is “shadowed”) unless I check “allow package selection”. After checking this option the dtBase2 file converts to a folder and is actually syncing the folder over to my iMac. In my iMac I only see the folder (and of course its contents), but not the dtBase2 file. Any solutions?

DEVONthink databases are folders with .dtbase2 extensions. If the extension is lost, merely add it back and you’ll once again see the whole package as a database.

Now the sync works nice, I had to choose the folder that stores the database, not the database per se. From my MacBook I changed the location of a tag and did a test-sync to the iMac, the sync took about 2 seconds and worked fine (used the dissect packages option in Chronosync).

That’s normally what I do, also. In order to ensure that my indexed files work as expected on all machines, I have a folder structure like this:

~/Documents
~/Documents/Work Papers
~/Documents/Work Papers/Databases
~/Documents/Work Papers/Indexed Documents

I put the databases in ~/Documents/Work Papers/Databases and I put the indexed documents in subfolders of ~/Documents/Work Papers/Indexed Documents. It is the ~/Documents/Work Papers folder that I sync.

This way, regardless of the name of the machine or its active user name, the indexed documents are always placed at the same position in the document hierarchy relative to the root of the active user. And so, indexed documents always work. The same would be true if you put indexed documents in that user’s Dropbox, or Box, or Google Drive, or Amazon Cloud Drive, or Adobe Creative Cloud folder, or WebDAV, or …

I’d advise reading @arnow’s post about dissect packages, upstream on this thread

This is a very informative thread. Thanks to all who are participating.

I am trying to extract the best practice from all of the information presented. This is what I think the best practice might be:

  1. Make sure that your 'think database is closed before attempting to sync.

  2. Use 'think’s database verification utility periodically to ensure that your databases are healthy and to ensure that you do not propagate unhealthy databases between computers.

  3. If you use indexing for any databases, you must ensure that the identical file and folder hierarchy exists on both computers, and sync the 'think databases and indexed content as one “logical transaction” to ensure that all changes are synced. Do not make changes to indexed databases until they and their underlying content stores are successfully synced.

  4. Most errors seem to be caused by making changes to both copies of a database and trying to use the more efficient “dissect packages” option, and syncing bidirectionally. To make syncing the most efficient and to protect databases’ content and integrity, use “dissect packages” but make changes to only one copy of the database prior to syncing. In this way, there is no need to sync bidirectionally, which risks problems with database integrity because of unresolvable conflicts between the copies of the databases.

  5. Make sure that you have good backups of your databases from all of your computers. Syncing is not a backup strategy, but a way of making your databases available on different computers.

Note that any AppleScripts that you use in conjunction with any database, whether called from 'think’s Scripts menu, attached to a Group as a triggered script, or added to 'think as a Template, will not be synced simply by virtue of syncing a database. This is because scripts are not stored in databases. Scripts must be synced or copied between computers separately.


Please correct and add to the list. Thanks again for a very informative thread.

PS: arnow, what is the “administrative file” that you refer to? Is this something that ChronoSnyc creates and maintains?

@Shoolie – very nice summary. I wouldn’t change anything you suggested (except my personal preference is not to use “dissect packages” because I’m comfortable with copying the whole database – I’m not in such a hurry that it matters to me) 8)

It is something that DEVONThink creates and maintains!

To explain what I mean I need to take you on a little expedition.

Make a copy of one of your DEVONThink databases or create a test database with a few documents. Make sure DEVONThink is closed.

In the Finder look at the database you just created (say test.dtbase2). It looks like a file. As I said above technically it is a bundle: a folder, that should be treated as a file. Now, control-click on the database and choose ‘Show package content’ in the menu that appears (the ‘contextual menu’). This gives you a view on the content of the bundle (I have no idea why Apple sometimes uses the term ‘package’ instead of ‘bundle’). You will see one or more Backup folders, ten .dtMeta files (DEVONthink-1.dtMeta, DEVONthink-2.dtMeta, and so on), a folder called ‘Files.noindex’, and a settings.plist file (there might also be other files or folders but they aren’t relevant to what I want to explain).

The Files.noindex folder contains the documents in your database, the .dtMeta files are what I called the ‘administrative’ files. You cannot open them. They contain the index of the database, a description of the group hierarchy in your database, lists that tell DevonThink which documents have which tags and so on, in a way understandable by the DEVONThink program, but not by humans.

Next open the Files.noindex folder. It contains folders for different types of documents (HTML, jpg, pdf, rtf, rtfd) etc. Each of these folders contains 16 subfolders numbered 0 to f. If you open these subfolders you’ll find the documents you added to your database (if you just created a small test database many folders are empty).You can open these documents if you like. Note that they do not open in DEVONThink, but in the program associated by the Finder to the relevant type of file: HTML files open in Safari, PDFs in preview and so on. This is because the Finder doesn’t know that these files are part of a DEVONThink database (neither does ChronoSync - I’ll return to that).

Note also, that the structure of the Files.noindex folder doesn’t mirror the group hierarchy of your database. That is why it is important to distinguish between groups in a DEVONThink database and folders in the filesystem. That is also one of the reasons why DEVONThink needs what I called ‘administrative files’: without these files it can’t construct the group hierarchy you see when you open a database in DEVONThink. Finally, it is one of the reasons why a DEVONThink database should be treated as a file rather than a folder: the database is the unit that makes sense.

End of the expedition.

I now explain the difference between ChronoSync sync and DEVONThink sync.

ChronoSync has no idea about DEVONThink and doesn’t know that a certain bundle contains data, it only knows about files and folders. During a sync session ChronoSync looks which folders and files have changed since the previous session and replaces, adds and deletes files accordingly (as I explained above it depends on the dissect packages setting whether a bundle such as a DEVONThink database is treated as folder or a file). That’s why I said that ChronoSync does file syncing.

DEVONThink Sync on the other hand does know about DEVONThink and treats the database as an organized structure that contains different types of data rather than as a file or a folder. It uses the administrative files to extract the data it needs and puts them in a store (DEVONThink syncs a DEVONThink database with a store, ChronoSync syncs a source folder with a target folder). At the next sync it uses the store and its administrative files to update the database and the store. That’s why I said that DevonThink sync does data syncing.

The key is this: ChronoSync sync replaces either the entire database (when ‘dissect packages’ is unchecked) or individual files (when dissect packages is checked). Moreover, it treats all files equally (it cannot distinguish between administrative files and documents added to a database). The first option is time consuming, the second option tricky (as I explained above).

If I understand the too scanty information provided by DevonTechnologies well enough (and, again, I am not sure I have), DEVONThink sync, on the other hand, understands the difference between the documents in Files.noindex and the ‘administrative files’ and treats these files differently: it updates the administrative files when it replaces, adds or removes documents in Files.noindex. In this way DevonThink sync should be faster than ChronoSync sync without dissecting packages (because it replaces individual documents) and much more safe than ChronoSync sync with dissect packages (because it maintains the integrity of the database by updating its administrative files), though slower (because updating the administrative files takes time).

Perhaps the DEVONThink team can tell us whether I am right about this difference?